Gesture control systems with logical states

ABSTRACT

Systems and methods for gesture-based control are disclosed. In some embodiments, a system may include a wearable device configured to obtain physiological data and wrist location data. The system may be configured to detect a first gesture and transition to an object control state in which changes in the wrist location data and/or the physiological data are configured to control a movement of the object. In the object control state, the system may be configured to detect a first motion of a body part, generate a first command to move the object in a first command direction, detect a second motion of the body part, and generate a command to move the object in a second command direction. The system may detect a second gesture indicating an intention to cease control of the object, and transition out of the object control state.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Pat. Application serial No.17/682,371, filed Feb. 28, 2022, which is a continuation of U.S. Pat.Application serial No. 17/384,527, filed Jul. 23, 2021, now U.S. Pat.No. 11,262,851 issued Mar. 1, 2022, which is a continuation of U.S. Pat.Application serial No. 16/890,507, filed Jun. 2, 2020, now U.S. Pat. No.11,157,086 issued Oct. 26, 2021, which is a continuation-in-part of U.S.Pat. Application serial No. 16/774,825, filed Jan. 28, 2020.

This application is related to U.S. Pat. Application serial No.16/104,273, filed Aug. 17, 2018, pending, which is a continuation ofU.S. Pat. Application serial No. 15/826,131, now U.S. Pat. 10,070,799issued Sep. 11, 2018, which is a nonprovisional application of U.S.provisional Pat. application 62/566,674, filed Oct. 7, 2017, and U.S.provisional Pat. application 62/429,334, filed Dec. 2, 2016, all ofwhich are hereby incorporated by reference.

This application also is related to U.S. Pat. application serial No.16/055,123, filed Aug. 5, 2018, pending.

This application also is related to U.S. Pat. application serial No.16/246,964, filed Jan. 14, 2019, pending.

This application also is related to PCT Pat. application serial No.PCT/US19/061421, filed Nov. 4, 2019, pending, which is an internationalapplication designating the United States claiming priority to U.S. Pat.Application serial No. 16/196,462, filed Nov. 20, 2018, pending.

This application also is related to U.S. Pat. application serial No.16/737,252, filed Jan. 8, 2020, pending.

All of the foregoing are hereby incorporated by reference.

BACKGROUND

Most machines have a form of “user interface” through which a personinteracts with the machine. The person provides inputs through one ormore devices from which the machine interprets the person’s intent. Themachine provides feedback to the person in response to those inputs,such as by the behavior of the machine or by outputs through one or moredevices which present information to the person.

When the machine is or includes a computer system with a display, acommon paradigm for the user interface is a “graphical user interface”.With a graphical user interface, the person manipulates a user interfacedevice which provides input to an application running a computer. Inturn, the computer provides visual output on a display. The computer mayprovide other outputs, such as audio. Through the user interface device,the person can control a position of a graphical object, typicallycalled a cursor, within a display, and can indicate an action to beperformed. The action to be performed typically is based in part on theposition of that graphical object within the information presented onthe display. A variety of user interface devices have been created forcomputers. Likewise, many kinds of graphical user interfaces and otherkinds of user interface paradigms have been used with computers.

SUMMARY

This Summary introduces a selection of concepts in simplified form thatare described further below in the Detailed Description. This Summaryneither identifies features as key or essential, nor limits the scope,of the claimed subject matter.

The graphical user interface paradigm for interacting with a machine orcomputer can be difficult to use in some environments. In someenvironments, a computing device may have a small display or no displayat all. Further, in some environments, users may not be able to easilymanipulate any input devices, because they are grasping or holdinganother object. For a variety of reasons, in some environments, thetypical combination of an input device and a display does not provide asufficiently intuitive, workable, convenient, or safe form ofinteraction with a machine.

Further, most graphical user interfaces for a computer are based oninteraction of a user with graphical objects presented on a display orobjects otherwise known to and presented by the computer on the display.An input from a user tends to be associated with a location within thedisplay space known to and controlled by the computer.

As described herein, a human-machine interface is provided in which oneor more users provide one or more inputs indicative of a real-world,geographical location, from which a geographical location is determined.Each of the one or more users has a respective user interface devicefrom which a respective input is received. A responsive device receivesand processes an input from at least one user interface device. Theremay be multiple responsive devices, each receiving inputs from one ormore user interface devices. A responsive device, or another computingdevice in communication with the responsive device, can determine ageographic location based on one or more inputs received. Feedback canbe provided to at least one user regarding the determined geographiclocation.

Some example scenarios from which a geographic location can bedetermined include, but are not limited to, the following.

An individual may be pointing in a specific direction, such as north, orsouth, and may want information about what is in that general direction.An individual may point at a street corner or a building and may wantinformation about it, or simply to identify it. An individual may pointat a direction and may want to set a point on a map. An individual maypoint in a direction to identify an object, such as to spot a deer orother animal while hunting, or for communicating during a game or otheractivity.

Multiple individuals can be pointing at roughly the same location or asame object. Multiple users may want to cooperate to select or identifythe same object. Multiple users may want to cooperate to select oridentify multiple objects. The object or objects may be moving. Forexample, multiple firefighters can be pointing at a same window or otherposition on a building. Multiple individuals on a street can be pointingat the same vehicle on the street. For example, several police officerscan be pointing at the same car. Multiple individuals in a venue may bepointing at roughly the same location. Multiple people in a city parkmay be pointing at an object. For example, security personnel for avenue may be pointing at the same person, or at an open door, or at anobject. Multiple people on a search and rescue mission may identify anobject, or a person or an animal to be rescued, or a hazard or obstacle.

The selected geographic location can be used to control communication.For example, an individual can point at another individual and initiatecommunication directly with that individual. An individual can select agroup of individuals, such as by pointing at them individually orselecting them as a group with a gesture, to communicate with the group.An individual can, with a gesture such as pointing to the sky, initiatecommunication with a predetermined set of individuals. Similarly,communication with, or control of, or both, objects can be performed bypointing at an individual object, selecting a group of objects within aset, or selecting a predetermined set of objects.

The geographic location can be defined as an absolute location or arelative location. Examples of absolute locations include, but are notlimited to, coordinates in a map system, geographical positioning system(GPS) coordinates, or longitude and latitude in various resolutions, orany combination of these. Examples of relative locations include, butare not limited to, a reference point and a direction, or a referencepoint and a direction and a distance, a plurality of reference pointsand respective directions from those reference points, or a plurality ofreference points and respective directions and distances from thosereference points, or any combination of these.

The geographic location can be determined based on inputs from sensordata from a sensor associated with at least one person. Additionalsensor data from sensors associated from one or more other persons, orone or more other objects, or a combination of these, can be used todetermine the geographic location. Examples of such sensors include atleast any one or more of the following: a location sensor (as describedbelow), a geomagnetic sensor, a compass, a magnetometer, anaccelerometer, a gyroscope, an altimeter, an eye tracking device, anencoder providing information indicating a change in position, a cameraproviding image data or motion video, a reference to a known landmark,or sound sensor, such as one or more microphones, or other sensor thatcan provide an indication of a direction from a person, or anycombination of two or more of these. The sensor data can be interpretedas, for example, any of a number of primitive gestures by the person, anintended action, an arm angle or other indication of relative verticallocation, a direction or bearing, a distance, or an altitude or anycombination of these. The geographic location can be determined based onsuch sensor data received from sensors associated with one or morepersons.

In some implementations, the determination of the geographic locationcan be assisted using context data. Such context data can includeinformation, such as a geographic location, of one or more objects,voice, sound, or text data, image data from one or more cameras, mapdata or user inputs with respect to a map, or any combinations of theseor yet other context information.

In some implementations, additional information, such as such contextdata, can be used to provide inputs to select, control, actuate, orperform other actions related to a geographic locations or an objectselected based on the geographic location. For example, voice, gesture,or other data can be used as a modifier. For example, a firefighterpointing at a building can communicate through voice, e.g., “two personsare near the second floor window!”. For example, voice, gesture, orother data can be used to annotate a selected object, with differentgestures being used to provide different labels for the object, such as“safe” or “dangerous”. For example, voice, gesture, or other data can beused to activate a command with respect to an object, with differentactions being actuated in different ways. Such actuations can beanalogous to a “click”, but can be obtained from any other data receivedfrom a user.

The geographic location can be determined from the sensor data, andoptionally context data, using a variety of different kinds ofprocessing techniques. In some implementations, forms of triangulationcan be applied using directions from multiple reference points. In someimplementations, an input can allow a user to specify a distance inrelationship to a determined location and direction. In someimplementations, techniques from computational geometry can be applied,such as variants of simultaneous localization and mapping (SLAM)algorithms which compute a location of an agent, where the geographiclocation to be determined is treated as the location of the agent to becomputed. In some implementations, forms of odometry can be used wherechange in position over time is determined based on sensor data. In someimplementations, visual odometry can be performed on images fromcameras. Any combination of such implementations also can be used.

This geographic location in turn can be used in many ways, examples ofwhich include but are not limited to the following and can includecombinations of two or more of the following. A geographic location canbe used as an input to an application, such as to identify or select anobject. Various operations can be performed by an application inconnection with a selected object, of which non-limiting examplesinclude invoking tracking of the object by camera, or obtaininginformation about the object from a database. A geographic location canbe data communicated to another user or another machine. A geographiclocation can be treated as context information when selecting an actionto be performed in combination with data based on the signals receivedfrom one or more user interface devices. A geographic location can beprovided as feedback to a user, such as by displaying a point on a map.

In some implementations described herein, a human-machine interface,such as an interface for interacting with a computer orcomputer-controlled device, can be implemented by a combination of oneor more user interface devices worn by a person, and one or moreresponsive devices associated with one or more machines or the person.The machine can include, for example, any type of computing device orcomputer-controlled device. The person performs, or attempts or intendsto perform, actions, which are sensed by sensors in the user interfacedevice; the sensors output signals. The responsive device interprets anduses or acts on data received based on the output signals from sensorsin the user interface device to enable the person to interact with themachine. Feedback is provided to the person. Such feedback can beprovided by the machine, by the responsive device, or by the userinterface device, or by other feedback devices, or a combination of twoor more of these. As described in more detail below, one kind of actionthat can be performed by a responsive device, and for which feedback canbe provided, includes identifying a geographic location.

In some implementations, the user interface device is worn by a personon a body part and includes at least a biopotential sensor and alocation sensor. The body part on which the user interface device isworn is an appendage which persons typically are capable of moving withrespect to their torso. Also, at that appendage, biopotentials relatedto activity of muscles that control a body part can be sensed at theskin surface, where the muscle activity enables the person to cause, orintend to cause, certain poses, as explained herein, of the body part.

In the example of a wrist-worn user interface device, the user interfacedevice is constructed to be worn by a person so that at least onebiopotential sensor is placed at least at the top of the wrist, asexplained herein, to sense biopotentials at the top of the wrist. Thebiopotentials sensed at the top of the wrist are related to activity ofmuscles that control the hand and fingers. Any pose, as explainedherein, of the hand is a result of activation or relaxation of thesemuscles. Biopotential signals are generated in response to a personintending to form any pose of the hand; the pose of the hand might notbe formed in some cases, such as when the person has a neuromuscularcondition. The biopotential sensor has an output providing abiopotential signal indicative of the sensed biopotentials.

In the example of a wrist-worn user interface device, the locationsensor senses location, as explained herein, of the wrist with respectto a reference point and has an output providing a location signalindicative of the sensed location. Different kinds of location sensorsmay use different reference points.

The responsive device is a kind of machine, typically a computingdevice, which receives, from the user interface device, data based onthe biopotential signals and based on the location signals. Theresponsive device performs actions based on the received data. Thereceived data may be the output signals from the sensors, or may be datarepresenting information derived from processing these signals, such asdata obtained from filtering or transforming these signals, dataindicative of features extracted from these signals, pose or locationinformation derived from the foregoing, or data representing primitivegestures detected from these signals, or other data obtained fromprocessing these signals, or a combination of any two or more of these.

The responsive device is responsive to the received data to detectprimitive gestures in the received data, and to cause actions to beperformed based on the primitive gestures. A primitive gesture caninclude a first pose of the hand, or a first location of the wrist, orboth, at a first moment in time; and a second pose of the hand, or asecond location of the wrist, or both, at a second moment in timefollowing the first moment in time. A primitive gesture can include a“hold” gesture in which the second pose is the same as the first pose.

In some cases, the action to be performed is based on a sequence ofdetected primitive gestures. In some cases, the action is based onspecific detected primitive gestures. In some cases, the action is basedon the co-occurrence of two or more detected primitive gestures. In somecases, the action is based on the occurrence of one or more primitivegestures in a particular context.

Typically, the action to be performed is an operation performed by acomputing device. This operation typically is performed by anapplication running on the computing device. A computing device may haveseveral applications running. When interacting with multipleapplications through a user interface device, the computing device isinstructed to provide an input to a selected application, and that inputcauses the selected application to perform a selected operation. Theprimitive gestures detected in the data received from the user interfacecan be used to select the application, or select the input to beprovided to an application, or both.

As an example, the responsive device can direct inputs to a selectedapplication, and the selected application can perform an action inresponse to a detected pose of the hand occurring after a period of timewithout detected motion of the wrist. Conversely, while motion of thewrist is detected, the responsive device can ignore data based onsignals from the biopotential sensor. This behavior can ensure thatmotion of the wrist does not generate signals that interfere withdetecting a pose of the hand.

As an example, the responsive device can select an application based ona detected location of the wrist. Within the selected application, whilethe wrist remains in the detected location, the application can selectand perform an action based on at least a detected pose of the hand.This behavior can enable, for example, switching between differentapplications. For example, a person can lift the wrist above the headand lift the index finger to activate an application, and cause thatapplication to perform an action. Afterwards, a person can lower thewrist and lift the index finger to activate another application andcause that other application to perform another action.

As an example, the responsive device can direct inputs to a selectedapplication, and the selected application can select an action based ona detected location of the wrist. The selected action can be performedin response to a detected pose of the hand while the wrist remains inthe detected location. This behavior can enable, for example, selectinga menu item or selecting a value within a range of values. For example,an audio playback system can select a song from a playlist, or adjustvolume.

As an example, the selected application can perform an action inresponse to a detected pose of the hand, occurring after a period oftime without detected motion of the wrist, followed by a detected holdof the pose accompanied by detected motion of the wrist in a direction.This behavior can enable, for example, selecting of an object byencircling the object, selecting a next or previous item in a list,adjusting scrolling or sliding elements, and other actions. For example,a person can lift the arm, then lift and hold a finger, and then make acircle motion, to select an object.

As an example, the responsive device can use a detected pose of the handto initiate and terminate a mode of operation that performs operationsprimarily based on the location of the wrist. This behavior can enable,for example, enabling a mode of operation to control movement on anobject, such as a drone, based on motion of the wrist. For example, inresponse to a lift and hold of a finger, a remote control mode for adrone can be activated, in which wrist location controls motion of thedrone.

Operations also can be performed based on any other information fromother sensors on the user interface device. Operations also can beperformed based on any context information available to the responsivedevice.

Such behavior in a responsive device enables a wide range of userexperiences for interacting with a machine in a variety of contexts.When the pose of the hand used to interact with the machine can beperformed while a person is grasping or holding another object, such aslifting a finger, the user interface device enables the person tointeract with the machine while performing other activities. When thepose of the hand used to interact with the machine can be performeddiscreetly, or with minimal motion or movement, such as when a personhas the hand in a pocket, the user interface device enables the personto interact with the machine conveniently or safely or secretly inchallenging environments. When the pose of the hand or location of thewrist used to interact with the machine is intuitive, such as pointingand swiping, the user interface device enables the person to interactwith the machine intuitively.

Accordingly, in one aspect, an apparatus includes a sensor constructedto be worn by a person and configured to sense information indicative ofa direction intended by the person. The apparatus includes a processor,and storage for instructions executable by the processor. When executingthe instructions, the processor determines the direction based on theinformation sensed by the sensor, and, based on the direction and alocation of the person, determines a geographic location.

In another aspect, a responsive device is constructed to communicatewith a user interface device. The responsive device includes aprocessing system comprising a processing device and computer storage.The computer storage has computer program instructions stored thereonwhich, when processed by the processing device, configure the responsivedevice. The responsive device is configured to receive sensor data fromthe user interface device and determine a geographical location.

In some implementations, the apparatus includes a biopotential sensorand a location sensor. The biopotential sensor is constructed to be wornby a person and to respond to an intention expressed by the person. Thebiopotential sensor is configured to sense biopotentials appearing onthe person. The biopotentials are associated with muscles controlling abody part or controlling an appendage near the body part. The locationsensor is configured for sensing location on the person. A processor,with storage for instructions executable by the processor, detects aprimitive gesture representative of the intention expressed by theperson, based on the sensed biopotentials and sensed location. Inresponse to the primitive gesture, the processor determines thegeographic location.

In some implementations a responsive device is configured to receivedata based on a biopotential signal and data based on a location signal.The responsive device is configured to detect a primitive gesturerepresentative of the intention expressed by the person. In response tothe primitive gesture, the processor determines the geographic location.

Any of the foregoing can include one or more of the following features.The sensor comprises a biopotential sensor constructed to be worn by theperson and configured to sense biopotentials appearing on the person.The sensor comprises a location sensor constructed to be worn by theperson and configured to sense a location for the person. The sensorcomprises an eye tracking device. The sensor comprises a geomagneticsensor. The sensor comprises an altimeter. The sensor comprises anencoder providing information indicating a change in position.

Any of the foregoing can include one or more of the following features.The body part is the wrist. The appendage is the hand. The appendageincludes one or more fingers of a hand. The pose is a pose of the hand.The pose of the hand includes a pose of one or more fingers. The pose ofthe hand includes a movement of one or more fingers. The pose of thehand includes a pose of a single finger. The pose of the hand includesmovement of a single finger. The single finger is the thumb. The singlefinger is the index finger. The pose of the hand includes a position ofthe hand relative to the wrist. The primitive gesture comprises a fingerswipe. The primitive gesture comprises a finger lift. The primitivegesture comprises a finger hold. The location of the wrist comprises aposition of the top of the wrist. The location of the wrist comprises anorientation of the top of the wrist. The location of the wrist comprisesmotion of the top of the wrist.

Any of the foregoing can include one or more of the following features.The location sensor comprises an accelerometer that generates locationsignals comprising a signal indicative of direction and magnitude ofacceleration of the body part. The location sensor comprises a gyroscopethat generates location signals comprising a signal indicative oforientation and angular velocity of the body part. The location sensorcomprises a geomagnetic sensor that generates location signalscomprising a signal indicative of a heading.

Any of the foregoing can include one or more of the following features.The geographic location is determined based on a first geographiclocation of the person, a first direction based on at least the sensordata, and a second direction from a second geographic location. Thesecond geographic location is of a second person. The second directionis a direction indicated by the second person. The second geographiclocation is a location derived based on a signal from a device carriedby the person. The second direction is a direction derived based on asignal from a device carried by the person.

Any of the foregoing can include one or more of the following features.The geographic location is determined based on data indicating ageographic location of the person and a direction and a distance alongthe direction. The distance along the direction is based on at least oneof the biopotential signals or the location signals from the person. Thedetermined geographic location specifies a point on a map. In responseto a detected primitive gesture, the processor modifies the location ofthe specified point on the map.

Any of the foregoing can include one or more of the following features.The geographic location is determined based on a selected object. Inresponse to a geographic location of a person, and in response to adetected primitive gesture, an object having a geographic location isselected. The geographic location of the selected object is thedetermined geographic location. The geographic location of the selectedobject is based on accessing information about the geographic locationof the selected object from a database. The geographic location of theselected object is based on computing the geographic location of theselected object based on time of flight data of the object.

Any of the foregoing can include one or more of the following features.The location on the person comprises a position of the top of the wrist.The location on the person comprises an orientation of the top of thewrist. The location on the person comprises motion of the top of thewrist.

Any of the foregoing can include one or more of the following features.The processor determines the geographic location by receiving dataindicating a first geographic location of the person, determining afirst direction based on the sensor data received from the userinterface device worn by the person, determining a second direction froma second geographic location, and determining the geographic locationbased on the first geographic location and the first direction, and onthe second geographic location and the second direction. The secondgeographic location comprises a geographic location of a second person.The second direction comprises a direction indicated by the secondperson. The second geographic location comprises a location derivedbased on a signal from a device carried by the second person. The seconddirection comprises a direction derived based on a signal from a devicecarried by the second person. The second geographic location comprises alocation of a second sensor. The second direction comprises a directionbased on signals from the second sensor.

Any of the foregoing can include one or more of the following features.The processor determines the geographic location by receiving dataindicating a geographic location of the person and a direction, andbased on the sensor data, determining a distance along the firstdirection, and determining the geographic location based on thegeographic location of the person, the direction, and the determineddistance. The determined geographic location specifies a point incoordinates associated with a map. In response to a detected primitivegesture, the processor modifies the location of the specified point.

Any of the foregoing can include one or more of the following features.To determine the geographic location, the processor is configured to, inresponse to a geographic location of a person, and in response to thesensor data, select an object having a geographic location, provide thegeographic location of the selected object as the determined geographiclocation. The geographic location of the selected object comprisesaccessing information about the geographic location of the selectedobject from a database. The geographic location of the selected objectcomprises computing the geographic location of the selected object basedon time of flight data of the object.

Any of the foregoing may be embodied as a computer system, as acomponent of such a computer system, as a process performed by such acomputer system or a component of such a computer system, or as anarticle of manufacture including computer storage in which computerprogram code is stored and which, when processed by the processingsystem(s) of one or more computers, configures the processing system(s)of the one or more computers to provide such a computer system or acomponent of such a computer system.

The following Detailed Description references the accompanying drawingswhich form a part this application, and which show, by way ofillustration, specific example implementations. Other implementationsmay be made without departing from the scope of the disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a block diagram of a user interface device.

FIGS. 1B-1C are perspective views of an example user interface device.

FIGS. 2A-2D are block diagrams of example implementations of aresponsive device.

FIG. 3 is an illustrative schematic of processing stages of signals fromthe user interface device.

FIG. 4A illustrates example hand poses.

FIG. 4B illustrates example locations of the top of the wrist.

FIG. 4C is a state diagram illustrating processing of data received fromthe user interface device.

FIG. 5 is a data flow diagram illustrating interaction between the userinterface device and the responsive device to provide inputs toapplications on the responsive device.

FIG. 6 is a schematic diagram of how an application can select differentoperations based on the location of the wrist.

FIG. 7 is a state diagram of how an application can perform differentoperations based on a pose of the hand or a location of the wrist orboth.

FIG. 8 illustrates graphically how location of the wrist can be mappedto different operations.

FIG. 9A is a data flow diagram illustrating determining a geographiclocation.

FIGS. 9B and 9C illustrate an example scenario of determining ageographic location/

FIG. 9D is a data flow diagram of an application that selects anoperation to be performed on an object selected within athree-dimensional scene.

FIG. 10 is an example graphical user interface.

FIG. 11 is an example graphical user interface.

FIG. 12 is a block diagram of a general-purpose computer.

DETAILED DESCRIPTION

As described herein, a human-machine interface, such as an interface forinteracting with a computer or computer-controlled device, isimplemented by a combination of one or more user interface devices wornby a person, and one or more responsive devices associated with one ormore machines or the person. The machine can include, for example, anytype of computing device or computer-controlled device.

The person performs, or attempts or intends to perform, actions, whichare sensed by sensors in the user interface device; the sensors outputsignals. The responsive device interprets data received based on theoutput signals from sensors in the user interface device to enable theperson to interact with the machine. Feedback is provided to the person.Such feedback can be provided by the machine, by the responsive device,or by the user interface device, or by other feedback devices, or acombination of two or more of these. As described in more detail below,one kind of action that can be performed by a responsive device, and forwhich feedback can be provided, includes identifying a geographiclocation.

There are many possible configurations and interrelationships of userinterface device, responsive device, and machine, some of which aredescribed in more detail below in connection with FIGS. 2A-2D.Typically, the user interface device, responsive device, and machine maybe separate devices. Two or more of the devices may be integrated intothe same device, such as: a single device incorporating both a userinterface device and a responsive device (e.g., a smart watch controlsanother machine); a single device incorporating both a responsive deviceand a machine (e.g., a user interface device interacts with a smartphone or tablet or computer); and a single device incorporating the userinterface device, responsive device, and a machine (e.g., a smartwatch). The devices may be partially integrated into the same device,such as when functionality described herein as residing in a responsivedevice is located in a user interface device (as noted in more detailbelow in connection with FIG. 3 ). There may be multiple instances ofuser interface devices, or of responsive devices, or of machines, or ofcombinations of them. For example, in some cases, multiple userinterface devices may be worn by one person, with respective userinterface devices worn on respective parts of the body, such asdifferent wrists or ankles. In some cases, multiple user interfacedevices may be worn by multiple different people, such as a first personwearing a first user interface device and a second person wearing asecond user interface device, with both user interfaces connecting toone or more responsive devices. The connections among the devices can beimplemented in many ways, including wired and wireless communication,computer networks, computer bus structures, and the like, andcombinations of them.

FIG. 1A schematically illustrates a block diagram of a circuit in a userinterface device 100 worn on a person. The body part on which the userinterface device can be worn can be an appendage which persons typicallyare capable of moving with respect to their torso, for example, the armat the top of the wrist. Also, at this appendage, biopotentials relatedto activity or intended activity of muscles that control a body part canbe sensed at the skin surface, where the muscle activity enables theperson to cause, or intend to cause, certain poses of the body part.

While such a user interface device can be worn on many different bodyparts of the person, the user interface device will be described hereinusing an example of wearing the user interface device such that at leastone biopotential sensor is placed at the top of the wrist. The phrase“top of the wrist” as used herein is intended to mean the posteriorsurface of the arm at or near the distal end of the arm adjacent to thejoint (i.e., the wrist) connecting the hand to the arm (described inmore detail below in connection with FIGS. 4A-4C).

In some implementations, in the case where the user interface device isworn at the top of the wrist, the user interface device 100 outputs data120, including at least signals 102 indicative of location of the wristrelative to a reference point and signals 104 indicative ofbiopotentials related to activity of muscles that control the hand andfingers of the hand connected to the wrist. To produce these signals,the user interface device 100 has at least two sensors, a biopotentialsensor 106 and a location sensor 108. When the user interface is worn onanother body part on the person other than the wrist, the user interfacedevice will output signals indicative of the location for that body partwith respect to a reference point and biopotentials sensed at the skinsurface at that body part.

The phrase “location” is intended to mean any data describing, at leastin part, position or orientation with respect to a reference point, aheading, or changes in any of these, such as velocity, angular velocity,acceleration, or angular acceleration, or any combination of two or moreof these. Within a frame of reference, such as a coordinate systemassociated with a person’s body, the reference point may be absolute(such as an origin in a frame of reference) or relative (such as apreviously known location). Different kinds of location sensors may usedifferent reference points. Location may be represented in coordinates,in one, two, or three dimensions, whether in cartesian, radial, orspherical coordinates, or as a vector. Location also may be expressed inrelative terms, such as high, low, left, right, far, and near andcombinations of them. As used herein, “motion” of a body part such asthe wrist is intended to signify a change in the position ororientation, or both, of the body part over time with respect to thereference point. Typically, a change in position or orientation of thetop of the wrist is due to arm motion, and is referred to herein iswrist motion or arm motion. Other movements of the hand or finger,though possibly occurring at or near the wrist (or other body part),such as bending of the hand or extension or contraction of one or morefingers (or similar body part), are herein called “movement”.

“Biopotentials” are electrical signals propagating in the body which canbe noninvasively sensed on the surface of the skin and which areindicative of nerve signals transmitted to muscles, electrical activityof muscles in response to such nerve signals, and other electricalactivity within tissues. When sensed at the top of the wrist, suchbiopotentials are related to activity of the muscles that control thehand and fingers, including movement of a single finger or groups offingers with respect to the hand or to each other, and movement of thehand with respect to the arm, and any combination of these. The phrase“related to activity of muscles” is intended to include any one or moreof nerve signals, muscle tissue signals, signals related to intendedmuscle movement (whether or not actualized), or signals related tounintended muscle movement (whether or not actualized), or anycombination of two or more of these. The muscle movement may or may notbe actualized depending on the condition of the person. The person mayhave a disease, such as atrophic lateral sclerosis (ALS) in which themuscles do not move as intended or do not move at all, but nerve signalsstill can be generated for intended muscle movement. The person may beotherwise constrained, such that muscle movement does not occur, butnerve signals and possibly muscle tissue signals are generated. In someinstances herein, the text refers to movement of a body part, such asmovement of a finger. Those instances should be understood as a shortreference and include not only actual movement of the body part, butalso intended, but not actualized, movement of that body part, in whichcase the biopotential signal indicates intended muscle movement forcontrolling that body part.

The user interface device 100 also can output feedback to the personthrough one or more feedback devices 110, examples of which aredescribed in more detail below. The user interface device 100 mayreceive feedback signals 112 related to the feedback devices 110. Theuser interface device 100 may generate (e.g., by controller 130) thefeedback signals 112 for the feedback devices 100. The user interfacedevice 100 may not be the sole source of feedback to the user, as suchfeedback can be provided from the responsive device, the machine, orother feedback device. In some implementations, the user interfacedevice 100 may not have any feedback device. For example, a flashinglight may indicate that the user interface device is communicating withthe responsive device, or, a vibration may indicate that the responsivedevice has completed an action.

The user interface device 100 also may have one or more input devices114, such as a button, through which the person can provide inputsintentionally or unintentionally directly to the user interface device100.

The user interface device 100 typically communicates over a wirelessconnection with the responsive device. The user interface device may bepowered by battery. The user interface device, when separate from theresponsive device, can communicate with the responsive device (to senddata 120 and receive feedback signals 112) through a wirelesstransceiver (not shown).

The various components of the user interface device, e.g., 106, 108,110, 114, and 122, interact with a controller 130. The controller isconnected to these components to control their operation. The controllercan include an analog-to-digital converter that samples, downsamples, orupsamples the outputs of the sensors 106, 108, 122 to provide a sequenceof digital samples at a desired sampling rate from the output signals ofthe sensors. The controller 230 can include any other circuit thatprocesses or conditions the data 102, 104, and 124 prior to sampling,such as any amplifier or band pass, notch, high-pass or low-passfiltering.

The biopotential sensor 106 senses biopotentials at the location on thebody where the user interface device is worn, such as at the top of thewrist. The biopotential sensor 106 provides one or more output signals,i.e., biopotential signals 104, indicative of the sensed biopotentials.

The biopotential sensor 106 comprises one or more pairs of electrodes.Each pair of electrodes is called a “channel” herein, as described abovein connection with FIG. 1C. When the user interface device 100 is wornon the wrist, the electrodes are arranged to be in contact with the skinat the top of the wrist and the electrodes within each pair are orientedsuch that a first electrode is closer to the fingers than a secondelectrode of the pair. In some implementations, the kinds of electrodesand circuitry as described in related applications listed above, or inFIG. 1C above, can be used to provide each channel of the biopotentialsensor 106.

A differential signal can be generated from the outputs of each pair ofelectrodes. The differential signal can be generated from the outputs ofeach pair of electrodes by using an operational amplifier, such asdescribed in U.S. Pat. 10,070,799. In implementations using a referenceelectrode for additional noise rejection, such as electrode 180 in FIG.1C, the differential signal also can incorporate the signal from thereference electrode. For example, given a signal (e1) for a firstelectrode, a signal (e2) for a second electrode, and a signal (r) forthe reference electrode, the reference signal (r) can be firstsubtracted from the signals (e1, e2) of the first and second electrodes,which can be performed in the analog domain, before generating thedifferential signal for the pair of electrodes, such that thedifferential signal for the pair of electrodes can be expressed as:(e1-r) - (e2-r). The differential signal can be the output of thechannel. The outputs from one or more channels make up the biopotentialsignals 104.

The biopotential signals 104 are used to detect poses of the hand, or apose of another body part near which the user interface device is worn.A “pose of the hand” includes a position or a movement of one or morefingers relative to the hand, or a position or a movement of the handrelative to the wrist, or any combination of these. Accordingly, asystem is designed and implemented, in a relatively simple case, todetect at least a first pose, and distinguish that first pose from thelack of any pose. If a second pose is introduced, then the system isdesigned to distinguish the first pose from the second pose, and thefirst and second poses from the lack of either pose. The biopotentialsensor 106 therefore is designed based on the application, such thatbiopotentials sensed by the biopotential sensor 106 carry sufficientinformation to enable detection of a specific pose by the person bothcompared to noise and compared to other specific poses to be detected.

In applications where the user interface device is worn at the top ofthe wrist, the biopotential signal may be processed to detect a pose ofthe hand. This processing can involve distinguishing between movement ofany finger and no movement of any finger. This processing can involvedistinguishing one kind of movement of one finger from a differentmovement of the same finger or from the same or different movement of adifferent finger. This processing can involve distinguishing one pose ofthe hand from another pose of the hand. The biopotential sensor can be asingle channel, comprising a single pair of electrodes. The biopotentialsensor can have only two channels, with each channel having a singlepair of electrodes. The biopotential sensor can have only threechannels. The biopotential sensor can have four or more channels. Theoutput signals from the biopotential sensor can be processed usinginformation from other sensors to help distinguish among different posesof the hand.

Within a single channel, larger spacing between the electrodes improvesthe ability of an output differential signal for that channel to providemore meaningful information. With two or more channels, larger spacingbetween the two channels improves signal separation and reducescrosstalk. The spacing of the electrodes (“electrode spacing”) and thespacing of the channels (“channel spacing”) are related to, and thusselected based on, characteristics of the biopotentials propagatingwithin the part of the body on which the user interface device isintended to be worn.

For some applications, each pair of electrodes can be positioned tooptimize capture of nerve signals. Capturing nerve signals can behelpful for applications where the person is known to have aneuromuscular condition such as atrophic lateral sclerosis (ALS) orotherwise may be constrained from having actual muscle movement. Forexample, on the top of the wrist, electrode pairs can be placed toemphasize capture of nerve signals related to the median nerve, ulnarnerve, or radial nerve, or any combination of these. However, in manyimplementations, the placement of two differential channels on the topof the wrist with sufficient spacing provides sufficient informationwithout requiring careful placement related to a specific nerve.

In some applications, the signal acquisition from the electrodes caninclude a form of dynamic impedance matching, such as described in U.S.Pat. 10,070,799. The operational amplifier providing the differentialsignal of the two electrodes in the channel can be connected to avariable resistor which is adapted to match the impedance of the skin.Without dynamic impedance matching, the sensor still may naturally adaptto the impedance of the wearer’s skin after a few minutes of wearing.However, implementing dynamic impedance matching in hardware may adaptthe sensor to the person more quickly.

As described in U.S. Pat. 10,070,799, the biopotential signal caninclude portions of signals from a variety of sources, including neural,neuromuscular, muscular, and external sources (such as radio waves).Some processing can be performed to reduce the influence of unwantedsignals, such as by using notch filters and band pass filters.Typically, external noise tends to be at 60 Hz and integer multiplesthereof, so notch filters for these different frequencies can be used.Muscle contraction electrical signals have a frequency range of 20 Hz to300 Hz with dominant energy in the 50 Hz to 150 Hz range. Nerve signalstypically have a frequency range of about 200 Hz to about 1000 Hz. Oneband-pass filter can be applied to attempt to focus on nerve signals.Another band-pass filter can be applied to attempt to focus on musclesignals. A band pass filter for frequencies of 20 Hz to 1000 Hz can beused to focus on both nerve and muscle signals. As described in moredetail below, such filtering can be performed by the biopotential sensor106 or another component on the user interface device, or can be part ofsignal processing performed by another part of the system.

As noted above, a location sensor 108 senses location of part of thebody where the user interface device is worn, such as at the top of thewrist, with respect to a reference point. When sensed at the top of thewrist, the sensed location is indicative of the location of the wristrelative to a reference point. The reference point can depend on thekind of location sensor. Examples include, but are not limited to, anabsolute point, such as an arbitrary reference point with respect to thebody, or the head or mouth or other specific body part, or a relativepoint, such as a previously sensed location. The location sensor 108provides an output signal, e.g., location signals 102, indicative of thesensed location. The concept of location of the body part, with awrist-worn user interface device, is explained in more detail below inconnection with FIG. 4A.

As an example, the location sensor 108 may include one or more of anaccelerometer, a gyroscope, a geomagnetic sensor or other form ofmagnetometer, or combination of two or more of these. For example, aninertial measurement unit or IMU can be used, which includes at least anaccelerometer and gyroscope, but also may include a geomagnetic sensor.An example of a commercially available product which can be used as thelocation sensor 108 is a 9-axis sensor combining three types of devices(accelerometer, gyroscope, and geomagnetic sensors), such as a BoschBNO055 absolute orientation sensor. Such a sensor can be used to providesensor data for determining a geographic location.

An accelerometer comprises a device that measures its own accelerationalong one or more axes and provides an output signal in response tostress or other force induced by the acceleration within the device.Given an initial position and acceleration data from an accelerometer inthree axes, a processing device can determine a position and orientationof the wrist in a three-dimensional space relative to its initialposition. Such a sensor can be used to provide sensor data fordetermining a geographic location.

A gyroscope comprises a device that measures orientation and angularvelocity along one or more dimensions. Given signals from a gyroscope inthree dimensions, one can determine orientation of the wrist in athree-dimensions relative to an initial orientation. Such a sensor canbe used to provide sensor data for determining a geographic location.

A geomagnetic sensor comprises a device that measures the earth’smagnetic field to provide a signal indicative of the measured magnitudeand orientation of the field. Given signals from a the geomagneticsensor, a processing device can determine a location, e.g., a directionor heading relative to a magnetic pole of the earth. Such a sensor canbe used to provide sensor data for determining a geographic location.

The location sensor 108 can include one or more sound detection devices,such as a single microphone or a plurality of microphones. The sounddetection device can be a microphone array including a plurality ofmicrophones spaced apart on the user interface device 100. The sounddetection device can be a microphone and resonant device on the userinterface device 100. The sound detection device outputs audio signals.The audio signals can be processed to determine the location of thewrist with respect to a location of a sound source, such as the mouth.An example implementation of such is found in co-pending patentapplication entitled “A Wearable Device Including a Sound DetectionDevice Providing Location Information for a Body Part”, by Dexter Angand David Cipoletta, Serial No. 16/737,242, filed on Jan. 8, 2020, andhereby incorporated by reference. Such a location can be computed usingsound source localization techniques. For example, the sound source canbe a mouth of the person wearing the user interface device. Sounddetection devices can be used in combination with any one or more otherkinds of location sensors. Sound detection devices can be used toprovide sensor data for determining a geographic location.

The user interface device also can include yet additional sensors (e.g.,122) of one or more other types which output other sensor data 124.Examples of types of sensors that can be used as the additional sensorsinclude, but are not limited to, any device that can sense, detect, ormeasure a physical parameter, phenomenon, or occurrence, such as globalpositioning system (GPS) sensors, moisture sensors, temperature sensors,visible light sensors, infrared sensors, image capture devices such ascameras, audio sensors, proximity sensors, ultrasound sensors, radiofrequency signal detectors, skin impedance sensors, pressure sensors,electromagnetic interference sensors, touch capacitance sensors, andcombinations of them.

A geographic location can be determined based on inputs from sensor datafrom a sensor associated with at least one person. Additional sensordata from sensors associated from one or more other persons, or one ormore other objects, or a combination of these, can be used to determinethe geographic location. Examples of such sensors include at least anyone or more of the following: a location sensor (as described below), ageomagnetic sensor, a compass, a magnetometer, an accelerometer, agyroscope, an altimeter, an eye tracking device, an encoder providinginformation indicating a change in position, a camera providing imagedata or motion video, a reference to a known landmark, or sound sensor,such as one or more microphones, or other sensor that can provide anindication of a direction from a person, or any combination of two ormore of these. The sensor data can be interpreted as, for example, anyof a number of primitive gestures by the person, an intended action, anarm angle or other indication of relative vertical location, a directionor bearing, a distance, or an altitude or any combination of these. Thegeographic location can be determined based on such sensor data receivedfrom sensors associated with one or more persons.

In some implementations, a user interface device can include a sensorfor sensing data for determining a geographic location withoutbiopotential sensors or location sensors. A person can be wearingmultiple user interface devices, with different user interface deviceshaving different sensors providing different data.

Turning now to the feedback devices 110, any device (or combination ofdevices) that provides feedback to the person wearing the user interface100 can be used. Feedback is any stimulus which is perceptible to theperson and which results from the action performed by the responsivedevice. For the purposes of illustration in FIG. 1A, the user interfacedevice 100 is illustrated as including a feedback device 110. However,in some implementations, a feedback device may be present outside of theuser interface device 100, instead of or in addition to any feedbackdevice 110 in the user interface device 100.

As an example, the feedback device 110 can include a source of hapticfeedback, also called a haptic device. A haptic device has physicalcontact with the person, typically by noninvasive contact with the skin,whether directly or indirectly through some material. The haptic devicegenerates a stimulus that can be perceived by the person, typicallythrough the skin (tactile stimulus) or through the sense of motion andposition of the limbs (kinesthetic stimulus). Examples of hapticfeedback can include, but is not limited to: vibration, motion, force,pressure, acceleration, heat, texture, shape, or other forms of tactileor kinesthetic sensations. The user interface device 100 can receivedata from an external source, as indicated by feedback signals 112, forcontrolling the state of the haptic device. The user interface device100 can generate or can have stored data for controlling the state ofthe haptic device, through the controller 130.

As another example, the feedback device 110 can include a light source,such as one or more light emitting diodes. For example, a singlelight-emitting diode can be placed on a visible surface, from theperspective of the wearer, of the user interface device 100. Of course,multiple light-emitting diodes can be used. The light-emitting diode canproduce light of a single color or of multiple colors. Thelight-emitting diode can produce light of a single intensity or multipleintensities. The user interface device 100 can receive data from anexternal source, as indicated by feedback signals 112, for controllingthe state of the light source. The user interface device 100 cangenerate or can have stored data for controlling the state of the lightsource, through the controller 130.

As another example, the feedback device 110 can include a sound source,such as one or more speakers. For example, a small speaker can be placedon a surface directed toward the person wearing the user interfacedevice 100. Of course, multiple speakers can be used. The speaker can beselected based on the desired sound signals to be produced, such as asingle tone or frequency, up to a full range of audible sounds, such asmusic. Sounds can be produced at a single set volume or at varyingvolumes. Volume can be adjustable by the person, the user interfacedevice, or other device (such as through the feedback signal 112). Theuser interface device 100 can receive data from an external source, suchas indicated by feedback signals 112, for controlling playback of soundon one or more speakers. The user interface device 100 can generate orcan have stored data for controlling playback of sound on the one ormore speakers, through the controller 130. Such feedback signals caninclude audio data, or control data, or both.

As another example, the feedback device 110 can include a display, suchas an array of N pixels by M pixels which display images. For example, asmall display can be placed on a surface directed toward the personwearing the user interface device 100. The display can be selected basedon the desired resolution of images to be produced on the user interfacedevice, which may depend on the application, and can support simple textor graphics, up to full-motion video. Controls may be available toadjust the contrast, brightness, color balance and other characteristicsof the display, which may be adjustable by the person, the userinterface device, or other device (such as through the feedback signal112). The user interface device 100 can receive data from an externalsource, such as indicated by feedback signals 112, for controllingdisplay of content on the display. The user interface device 100 cangenerate or can have stored data for controlling display of content onthe display. The feedback signals 112 can include image data, text data,graphics data, video data, or control data, in any combination.

The user interface device 100 also may have one or more input devices114 through which the person can provide inputs to the user interfacedevice 100. While such input devices are, in some sense, sensors (suchas a sensor 122 mentioned above), an input device 114 is intended tomean a device which a person actively manipulates in order to provide aspecific input directly to the user interface device, such as a buttonor touchscreen. Such input devices typically would not produce signalsthat would be processed as part of the human-machine interface, butinstead are for controlling the user interface device, such as poweringon the device, initiating network connections, and controlling settings.

In an example configuration for a wrist worn user interface device,illustrated in FIGS. 1B-1C, a housing 150 contains the circuit shown inFIG. 1A, as well as its corresponding mechanical components. FIG. 1B isa top perspective view; FIG. 1C is a bottom perspective view. Fixtures152 allow a wrist band or other device (not shown) to connect to thehousing 150 to allow the housing to be worn. Buttons 154 are examples ofinput devices 114 (FIG. 1A); Light emitting diodes (LEDs) 156 areexamples of feedback devices 110.

As shown in FIG. 1C, an electrode 160 generally is made of stainlesssteel or other conductive material. In some implementations, theelectrode can be flat disc. In some implementations, the electrode canhave a surface with a curvature, such as a conical shape. The diameterof a round electrode is typically greater than 2 mm and less than 10 mm.In some implementations, the electrode can have a diameter of about sixmillimeters (6 mm).

Within a channel, e.g., a pair of electrodes 160 and 162, the electrodeshave an electrode spacing measured between the centers of theelectrodes. The electrode spacing 164 can be in the range limited on thelow end by the size of the electrodes and a desired spacing betweenthem, e.g., about 15 millimeters (15 mm) where the electrodes are 6 mmin diameter and about 9 mm of material is provided between the nearedges of the electrodes, and limited on the high end by the size of thedevice, e.g., about 25 millimeters (25 mm) where the electrodes are 6 mmin diameter and the device is about 30 mm. Generally, a maximum possiblespacing is desired, since decreasing electrode spacing results in alower signal amplitude. In some implementations, the electrode spacingcan be about 25 millimeters (25 mm) apart.

With two or more channels, pairs of channels have a channel spacingmeasured between a first line drawn through electrodes, e.g., 160 and162, of a first channel and a second line drawn through electrodes,e.g., 166 and 168, of a second channel. The channel spacing 170 can bein the range of about 10 millimeters (10 mm) to 20 millimeters (20 mm).The range is limited by anatomy on the high end and thus depends on thebody part on which the device is worn. With the top of the wrist, thehigh end of the range of channel spacing is about 20 mm. Decreasing thechannel distance generally results in lower source separation betweenchannels. In some implementations, the channel spacing can be about 15millimeters (15 mm). In some implementations, the biopotential sensorcan include one or more reference electrodes 180, which can be used fornoise reduction, as described in more detail below.

Turning now to FIGS. 2A-2D, several kinds of responsive device will nowbe described. A responsive device 200 is a kind of machine, typically acomputing device, which may be associated with one or more machines orone or more other computing devices, or both. The responsive device isresponsive to the user interface device 100 to allow the person tointeract with a machine, which may be the responsive device itself, ormay be a machine or computing device associated with the responsivedevice.

As shown in FIG. 2A, the responsive device 200 receives data 202 basedon the biopotential signals 104 and location signals 102 generated bythe user interface device 100. As described in more detail below, thereceived data 202 may be the output signals 102 and 104 from thesensors, or may be data representing information derived from processingthese signals, such as data obtained from filtering or transformingthese signals, data indicative of features extracted from these signals,data representing pose or location of a body part derived from thesesignals, or primitive gestures detected from these signals, or otherdata obtained from processing these signals, or any combination ofthese.

The responsive device 200 performs actions based on the received data202. The responsive device 200 also can generate feedback 204 to theperson, such as by the behavior of the responsive device or the machineassociated with it, or through the user interface device 100 by sendingfeedback signals 112 (FIG. 1A) to the user interface device, or throughother feedback devices 206 controlled by the responsive device 200. Asdescribed in more detail below, the actions to be performed, and thefeedback provided to the user, depend on the application, computingdevice, machine, or other system with which the person is interactingusing the user interface device and responsive device. Some examples ofsuch interaction will now be described in connection with FIGS. 2B-2D.

In FIG. 2B, the responsive device 200 is a computing device with whichthe person interacts. For example, the responsive device can be acomputer, such as a general-purpose desktop or laptop computer, smartphone, tablet, game console, set-top box, smart television, or the like.Some computing devices, such as a smart watch, can incorporate both theuser interface device and the responsive device within one physicaldevice. In a responsive device 200 in FIG. 2B, the computing devicetypically has an operating system 208 that supports running one or moreapplications 210. The user interface device 100 is used by the person tointeract with the operating system or the application(s). Feedback 204from the operating system or the application, typically in the form ofimages and sound, may be provided through displays, speakers, or otherfeedback device 206 accessible to the computing device.

In implementations of a configuration such as shown in FIG. 2B, the userinterface device can enable the person to interact with a variety ofapplications running on the responsive device, for example, a mediaplayback application, a camera, a map application, a weatherapplication, a search engine, an augmented reality application, avirtual reality application, a video game, or other application.

In implementations of a configuration such as shown in FIG. 2B, therecan be multiple different applications with which the user can interactthrough the user interface device. In such an environment, selectionamong these applications is an action, among others, that the responsivedevice can perform in response to the user interface device.

In FIG. 2C, the responsive device 200 is a computing device running aremote control application 212 that interacts with a machine 220 servedor controlled by the computing device. The person interacts with themachine 220 through the responsive device by using the user interfacedevice; in response to data received from the user interface device, theresponsive device sends inputs to the remote control application 212.The behavior of the machine 220 is the primary feedback of the system,but other feedback can be provided by the computing device or the userinterface device. For example, the remote control application 212 mayprovide other feedback 204 to a feedback device 206, such as a displayof the computing device or to the user interface device.

As an example, the other machine 220 may be a drone, also called anunmanned autonomous vehicle, for which the application 212 includes aremote controller. The application 212 generates control commands whichare transmitted to the drone by the computing device under control ofthe application 212. When the person is using the user interface device,the application can generate these control commands in response to thereceived data 202 based on the signals generated by the user interfacedevice. For example, the application can activate a remote controlsession in response to detection of a primitive gesture associated withthis function. During the remote control session, locations of the wristare interpreted by the application as providing directional controls forthe drone. The application can end the remote control session inresponse to detection of a primitive gesture associated with thatfunction.

As another example, the other machine 220 may be another computer, orcollection of computers. As another example, the other machine 220 maybe an articulated mechanical arm, which may or may not be considered arobot, such as used in manufacturing, telemedicine, or otherapplication. As another example, the other machine 220 may be a mobilerobot capable of traversing land or water, having a motor and a form ofpropulsion driven by that motor.

In FIG. 2D, the responsive device is a computing device 200 with acommunication application 230. The communication application 230 cansend messages 232 to the person through a feedback device 206. When theperson is using the user interface device, the communication application230 can send the messages 232 to the user interface device 100. Thecomputing device 200, or another computing device (not shown), may inturn receive data 202 from the user interface device. The received data201 may be responsive to the message 232, and, in the example shown inFIG. 2D, can be directed to the communication application. Feedback 204also may be provided for a number of purposes, such as to indicate thatthe computing device 200 received the data 202. Such a communicationapplication can have many forms.

In the example shown in FIG. 2D, the communication application 230 isused to gather labeled data for use in training a machine learningsystem 250. The responsive device sends a message instructing the personto manipulate the user interface device, for example by performing aspecified action. In turn, the responsive device receives data from theuser interface device. The responsive device can label that data ascorresponding to the manipulation the person was instructed to performand provided labeled data to the machine learning system 250.

As another example, to monitor the well-being of the person wearing theuser interface device, the responsive device may send a messagerequesting the person to respond. The responsive device then receivesdata from that person’s user interface device. Based on the receiveddata, the responsive device can perform additional actions, for examplethrough a monitoring application 260, such as notify a care providerabout the status of the person. An example implementation of such asystem is described in co-pending PCT Patent ApplicationPCT/US19/061421, entitled “Communication Using Biopotential Sensing andTactile Actions”, filed Nov. 14, 2019, which is hereby incorporated byreference.

As another example, an incoming message in a messaging application, suchas a voice call or text messaging application, can cause the responsivedevice to transmit a notification message to the user interface device.The responsive device then receives data from that person’s userinterface device. The responsive device can cause an action to be takenin the messaging application, such as answer a call or respond to atext, in response to the received data from the user interface device.

In any of such implementations of the user interface device 100 andresponsive device 200, the signals 102 and 104 generated at the userinterface device are transformed into inputs used by the applications onthe responsive devices. Such transformation may be performed by one ormore processing devices, which may reside on the user interface device,on the responsive device, or on both the user interface device and theresponsive device. In some implementations, such transformation may beperformed by one or more processing devices that reside on a computingdevice separate from the user interface device and from the responsivedevice.

In any of the foregoing configurations, one kind of action that can beperformed by a responsive device, and for which feedback can beprovided, includes identifying a geographic location. This geographiclocation in turn can be used in many ways, examples of which include butare not limited to the following and can include combinations of two ormore of the following. A geographic location can be used as an input toan application. A geographic location can be data communicated toanother user or another machine. A geographic location can be treated ascontext information when selecting an action to be performed incombination with data based on the signals received from one or moreuser interface devices. A geographic location can be provided asfeedback to a user.

The transformation of the signals 102 and 104 from the sensors 106 and108 into inputs to applications can involve several stages ofprocessing. Referring now to FIG. 3 , such stages of processing will nowbe described in further detail.

The first stage of processing is performed by sensors (300) in the userinterface device. Some sensors output digital signals. Some sensorsoutput analog signals. In some implementations, the biopotential sensorfor a channel may include a dynamic impedance matching circuit andamplifier to output an analog differential signal. The output of thisfirst stage is raw sensor signals.

The next stage 302 of processing can involve conditioning of the signalsoutput from the sensors. After any conditioning, the signals output fromthe sensors are sampled and synchronized. For sensors that output analogsignals, the analog signals may be subject to signal conditioningoperations, such as filtering and amplification, performed by analogcircuits. Analog signals are converted to digital signals using ananalog-to-digital converter. For sensors that output digital signals, orfor digital signals output from an analog-to-digital converter, thedigital signals may be subject to signal conditioning operations, suchas filtering and amplification, performed by digital circuits. Whetherthe conditioned signals are analog signals or digital signals, thesignals from various sensors can be sampled at the same sampling rate toprovide an output stream of digital sensor signals at the sampling rate,such that data from the various sensors are synchronized and at the samesampling rate. Such sampling can be performed by an analog-to-digitalconverter, or a digital-to-digital converter, and can include eitherupsampling, with or without interpolation, or downsampling, with orwithout filtering. The output stream of digital signals at the samplingrate from this second stage 302 is called herein “normalized digitalsensor signals”.

The normalized digital sensor signals can be subjected to some digitalsignal processing (304), for any filtering or other conditioning notapplied prior to sampling which can be applied after sampling. Forexample, the biopotential signals can be filtered to remove or reducethe effects of electrical interference, by applying a notch filter. Forexample, the biopotential signals can be filtered to remove or reduceeffects of other electrical activity affecting the body, to focus onlyon signals likely to be related to muscle movement, by applying a bandpass filter. If the biopotential signals are to be processed with anemphasis on the nerve signals, then the biopotential signals can befiltered to remove or reduce the effects of signals associated withmuscle activation, by using band pass filtering. Any other filtering orprocessing of signals derived from the outputs of any other sensors alsocan be performed in this digital signal processing stage 304. If used,the output of this third stage 304 is called herein “filtered”normalized digital sensor signals, which remains an output stream ofdigital signals at the sampling rate.

In the next stage 306, the optionally filtered, normalized digitalsensor signals, are processed to extract “features” of the signals. InFIG. 3 , this kind of processing is represented by a feature extractionlayer 306. A feature is any characteristic of a signal that can bequantified by applying an operation to the signal, or to atransformation of the signal (e.g., a frequency domain representation ofthe signal), or to a combination of two or more signals or theirtransformations. Typically, given a signal that is an output stream ofdigital signals at the sampling rate, features are computed for a streamof subsets of the samples, called windows, by applying a windowfunction. A window function can be specified by a moment in time in thestream of digital signals, and an interval. The result of applying thewindow function is that values are zero outside the interval. Thesimplest window function is a rectangular window which retains theoriginal samples within the interval. It is possible to use windowfunctions that apply a mathematical function other than the identityfunction to samples within the interval. For a sequence of windows of astream of digital samples, another parameter is the amount of overlapfrom one window to the next, i.e., the amount of offset from thebeginning of one window to the beginning of the immediately subsequentwindow. The offset can be the same as, less than, or greater than theinterval specified for the window. The output of this fourth stageincludes a stream of feature sets, with a feature set being output foreach window of the (optionally filtered) normalized digital sensorsignals. Several examples of features are described in more detailbelow.

In FIG. 3 , a next stage 308 of processing is called a “gesturedetection” layer. This stage involves processing the normalized digitalsensor signals, filtered normalized digital sensor signals, or streamsof feature sets extracted from them, to identify poses of a body part,location of a body part, or primitive gestures, or combinations orsubsets of these, collectively called “gesture detection”, in responseto which the responsive device causes an action to be performed. Notethat while FIG. 3 illustrates the feature extraction layer 306 betweenthe gesture detection layer 304 and the digital signal processing layer304, the gesture detection layer 308 can use, in some implementations,data output from the feature extraction layer, the digital signalprocessing layer, as well as the normalized digital sensor signals Inresponse to detection of a primitive gesture, the responsive devicedirects inputs, as indicated at 310, to the applications, in response towhich applications perform operations.

As shown in FIG. 3 , in some implementations, a pose, location, orprimitive gesture can be detected using a computational model called amachine learning model. The machine learning model is a model trained bya machine learning process using labeled data, an example of which isdescribed in more detail below. Typically, data from one or more userinterface devices, such as normalized digital sensor signals, filterednormalized digital sensor signals, or streams of feature sets extractedfrom those signals, are collected in a manner that also labels the dataas representing a known pose, location, or primitive gesture. A machinelearning (ML) training module 320 receives the labeled data andgenerates parameters for a trained machine learning (ML) model 322. TheML training module 320 can process this data outside of the real-timeinteraction between a user interface device and a responsive device. Thetrained ML model 322 is provided to the gesture detection layer 308,typically by providing parameters of the trained ML model to the gesturedetection layer 308, which applies the parameters within itsimplementation of the model. Thus, parameters for trained models areprovided to a component within the user interface device or a componentwithin the responsive device, such as the operating system, a devicedriver, or an application, which is responsible for performing gesturedetection.

Having now described the interaction among various processing layers,some example implementations for these different layers will now bedescribed.

At some times in these processing stages, information is transmittedbetween the user interface and the responsive device. This kind ofprocessing can be considered a signal transmission layer (not shown inFIG. 3 ), but, depending on the implementation, the signals transmittedfrom the user interface device to the responsive device on the signaltransmission layer may be the outputs of any of the layers 302, 304,306, or 308. In other words, the user interface device can transmitnormalized digital sensor signals, filtered normalized digital sensorsignals, a stream of feature sets derived from those signals, or pose orlocation information derived from these, or primitive gestures detectedbased on such data, or any combination on these. A suitable design forany specific implementation depends on how much processing is to beperformed within the user interface device before transmitting data tothe responsive device, and the information to be used on the responsivedevice to perform any further processing, which depends on designconstraints for the implementation.

Herein, this signal transmission layer is referred to as a deviceinterface. The device interface includes a specification of the typesand formats of information transmitted between the user interface deviceand the responsive device, a protocol of transmission of suchinformation, and a communication system over which the information istransmitted. In some implementations, the user interface deviceimplements a streaming data protocol to transmit a stream of digitalsamples based on the outputs of the biopotential and location sensors.The responsive device can send messages and feedback to the userinterface device on the device interface. Thus, the device interface canbe bidirectional. Alternatively, there may be a separate device feedbackinterface. Such a feedback interface may have its own specification ofthe types and formats of information transmitted between the responsivedevice and the user interface device, a protocol of transmission of suchinformation, and a communication system over which the information istransmitted. In some implementations, the feedback interface is providedby asynchronous messaging. In some implementations, the communicationsystem is a wireless Bluetooth connection between the user interfacedevice and the responsive device.

At some times in these processing stages, information also may becommunicated within the responsive device, from a first application(perhaps the operating system or a device driver for the user interfacedevice) running on the responsive device to a second application runningon the responsive device or on the machine associated with theresponsive device. In some implementations, the first applicationincludes the gesture detection layer, and the second application isdesigned to respond to outputs of the first application. In someimplementations, the second application may have a set of expectedinputs that can affect behavior of the application, and data receivedfrom the user interface device are mapped by the first application tothese inputs. In some implementations, the first application does notperform gesture detection, but instead passes data, such as digitalsamples or a stream of feature sets, to the second application, and thesecond application performs gesture detection.

Herein, the application interface refers to the information communicatedwithin the responsive device from the first application to the secondapplication (or more generally, to an application running on theresponsive device), and a protocol, including data format and messagingprotocol, for such communication. Because there may be outputs generatedby the second application, such as messages and feedback, to be sent bythe responsive device to the user interface device, this applicationinterface may be bidirectional. Alternatively, there may be a separateapplication feedback interface with its own data format and messagingprotocol.

In some implementations, such as a device driver (a first application)for the user interface device running as part of an operating system onthe responsive device, the first application can include both a deviceinterface, to handle interaction with the user interface device, and anapplication interface, to handle interaction with the secondapplication.

In some implementations, the first and second applications are combinedinto a single application, such that the single application implements adevice interface, to handle interaction with the user interface device,and otherwise processes the information from the user interface directlywithout intervention by another application.

There may be several application interfaces providing multiple layers ofprocessing. For example, a device driver (a first application) within anoperating system may process data received from the user interfacedevice into information about features detected, or poses detected, orlocation information, or primitive gestures detected; yet anotherapplication (the second application) may process such informationfurther, such as to map such information into corresponding events toemulate another kind of input device, and provide such events to yet athird application.

Within this context, a user interface device 100 (FIG. 1A) generateslocation signals 102 and biopotential signals 104, and a responsivedevice 200 (FIGS. 2A-2D) receives data 202 based on these signalsthrough a device interface. Within the responsive device, informationbased on the received data 202 can be passed to one or more applicationsthrough an application interface. The content of the data transmittedthrough the device interface and through the application interfacedepends on where the different stages of processing occur.

To reduce the amount of processing performed on the user interfacedevice when the user interface device is battery-powered and wirelesslyconnected to communicate data to the responsive device, the userinterface device can be configured to implement a device interface totransmit the normalized digital sensor signals. In this implementation,further processing of the data occurs on the responsive device.

In many computing devices that can be used to implement the responsivedevice, an operating system manages transfer of data from an inputdevice, such as the user interface device, to an application, typicallyusing a device driver. Some operating systems may allow data from aninput device, such as the user interface device, to bypass the operatingsystem and to be transferred directly to memory for an application. Thedevice driver can implement both a device interface to communicate withthe user interface device and an application interface to communicatewith an application, and the application implements the applicationinterface.

In implementations where the device interface provides the normalizeddigital sensor signals to the responsive device, a component on theresponsive device processes these signals to generate inputs forapplications. In some implementations, the device driver can passreceived data through to the application, in which case the applicationimplements the feature extraction and gesture detection layers. In someimplementations, the device driver can implement the feature extractionlayer, and the application can implement the gesture detection layer. Insome implementations, the device driver can implement both the featureextraction and gesture detection layers. In some implementations, afirst application implements part of the gesture detection layer, andpasses processed information to a second application.

The pairing of such a user interface device with such a responsivedevice supports many different contexts in which a person is enabled tointeract with a machine. Examples of such use contexts include, but arenot limited to, any one or more of the following.

The combination of the responsive device and the user interface devicemay have a very small display or no display at all. The person may beunable to use a display. The person may be focusing on an object andcannot simultaneously look at the object and a display. The person maybe grasping or holding another object. The person may be unable toprovide touch input to a touchscreen or similar device. The person mayneed to keep hands free to hold an object and not a user interfacedevice. The person may not be able to provide audible inputs. The personmay be unable to provide speech input. The person may be unable tospeak. It may be unsafe or undesirable for the person to speak or makenoise in their current environment. The person may not be able toreceive audible feedback. It may be unsafe or undesirable for the personto receive audible feedback in their current environment. The person maybe physically constrained or impaired, or otherwise may have limitedmobility. The person may be in a situation requiring limited perceptibleactivity, including motion or movement. In such context and in othercontexts, the person may be attempting to communicate or interact withother individuals in similar contexts.

To support such user-machine interactions in such contexts, theresponsive device is responsive to received data to perform operationsbased on the location signals and the biopotential signals generated onthe user interface device (as a result of movement of the fingers andhands and motion of the wrist and arm, for example) and on user inputsto the user interface device, data received based on other sensors onthe user interface device, and data about context from any source, andcombinations of them. In turn, the person receives feedback through oneor more of the user interface device, responsive device, machine, orother feedback device or combinations of them.

The operation to be performed also can be based on information from anyother sensors on the user interface device or based on other contextualinformation available to one or more of the user interface device, theresponsive device, or the machine. Examples of further contextualinformation that can be used to perform an operation include but are notlimited to the following and combinations of the following. Informationfrom a user interface device worn by another person can be used.Information from a sensor from another device, such global positioninginformation from a device associated with a person, can be used.Information accessed from a local or remote database can be used.

More particularly, primitive gestures are detected as occurring atmoments in time based on the location signals and the biopotentialsignals. In some cases, the operation to be performed is selected basedon occurrence of a primitive gesture. In some cases, the operation to beperformed is selected based on occurrence of a plurality of primitivegestures in a sequence. In some cases, the operation to be performed isselected based on co-occurrence of two or more primitive gestures. Insome cases, the operation to be performed is selected based on thecontext or contexts of such occurrences. Combinations of these bases forthe operation to be performed are also possible.

To illustrate kinds of primitive gestures that can be detected based onthe biopotential signals and the location signals, the followingdescription is based on an example of a user interface device worn onthe person’s wrist.

A primitive gesture may be characterized, as described in more detailbelow in connection with FIG. 4A, as a sequence of a first pose of thehand, or a first location of the wrist, or both, at a first moment intime; and a second pose of the hand, or a second location of the wrist,or both, at a second moment in time following the first moment in time.In some primitive gestures, such as a hold, the first and secondlocation are the same. A primitive gesture also can be considereddetecting such a transition, or absence of a transition.

A primitive gesture may be characterized by a set of features extractedfrom the location signals, or a set of features extracted from thebiopotential signals, or both, which differentiates that primitivegesture from other primitive gestures. The set(s) of features may beextracted from a single window of samples of the signal, or a sequenceof successive windows of samples of the signals. In someimplementations, the sets of feature(s) which characterize a primitivegesture correspond to one or more poses of the hand, or location orlocations of the wrist, or combinations of these.

A primitive gesture may be characterized by one or more states, asdescribed in more detail below in connection with FIG. 4C: an initialstate of a signal in a window just prior to the primitive gestureoccurring (a first set of features); an onset indicating a change in asignal in a window occurring at a beginning of an occurrence of theprimitive gesture (a second set of features); the states of or changesin a signal in one or more windows immediately after the onset (a thirdset of features); and a change in a signal in a window at the end of theprimitive gesture (a fourth set of features). A primitive gesture mayhave a moment in time at which the primitive gesture is considereddetected. A primitive gesture may be present for a duration of time, andthus may have a moment in time at which the primitive gesture begins anda moment in time at which the primitive gesture ends.

Because the output of the biopotential sensor (106 in FIG. 1A), whenlocated at the top of the wrist, is related to activity of muscles thatcontrol the hand and fingers, the biopotential signal is an importantsource of information about the pose of the hand. However, if thelocation, i.e., position or orientation, of the top of the wrist ischanging due to arm motion, such arm motion can cause the electrodes ofthe biopotential sensor to have inconsistent degrees of contact andlocations of contact with the skin, in turn causing inconsistentgeneration of the biopotential signals. Thus, the arm motion can reducethe ability to detect the pose of the hand from the biopotential signalswhen the arm is in motion. In such applications where this inconsistencyoccurs, data based on the biopotential signals can be ignored whilesignificant arm motion is detected. Herein, this operation of ignoringbiopotential signals is called “motion silencing”. Thus, in someimplementations, to detect a primitive gesture that is based on a poseof a hand, an initial state of a signal in a first window, just prior toa second window in which the pose can be detected, can be the absence ofsignificant changes in the location of the wrist. In the absence ofsignificant changes in the location of the wrist, the biopotentialsignals are analyzed to detect a pose of the hand.

While there many possible poses of the hand that can be detected, hereinare described several poses that relate to lifting one or more fingers.Motions of a finger can involved extension or contraction or both.Differences in biopotential signals generated for different primitivegestures of a finger can be used to differentiate the differentprimitive gestures. By processing the biopotential signals forindications of poses that relate to lifting one or more fingers, veryslight finger movements can be detected where those finger movements canbe performed while holding an object or while the hand is hidden, suchas in a person’s pocket.

Referring to FIG. 4A, a hand 450 is shown in a neutral position 452. Auser interface device 454 is shown as worn near the top of the wrist456. When a finger is lifted, as shown at 460, from this neutralposition 452, the first joint 458 closest to the hand is rotated towardthe back of the hand and away from the palm, in contrast to contractingthe finger towards the palm. Such movement also is distinct from a flickor a point in which the finger is extended or made straight withoutrotation of the first joint toward the back of the hand. Any finger, ormultiple fingers in sequence or together, can be lifted. Lifting afinger also can be understood as an action that lifts a finger away froma surface, such as a table top or object being held, while the palm ofthe hand is resting on that surface. While the hand remains in position,the finger is lifted away from the surface. The “onset” or second set offeatures characterizing this movement is a sudden increase in amplitudein the channels of the biopotential signal. Differences among thechannels of the biopotential signals allow lifting of the one finger,such as the index finger, to be distinguished from lifting of otherfingers.

A primitive gesture called a “lift” herein occurs when the finger islifted (460) for a short period of time, which can be defined by a fixedor adjustable threshold, and then returns to a resting state (e.g., 452)in which the biopotential signals no longer indicate that the finger islifted. Thus, the state of or change in the signals in one or morewindows immediately after the onset (the third set of features) is asudden drop in the biopotential signals related to the lifted finger.The lift can be considered detected upon detection of a small number ofwindows of elevated amplitude in the biopotential signal related to thelifted finger pose, followed by a number of windows for which thebiopotential signal related to the lifted finger drops significantly.The lift can be considered ended when detected and thus does not involvedetection of a fourth set of features to consider the lift ended.

A primitive gesture called a “hold” herein occurs when the finger islifted (460) and held for a period of time in that pose (460). Theperiod of time defining a lift versus a hold can be a fixed oradjustable threshold. Thus, the state of or change in the signals in oneor more windows immediately after the onset (the third set of features)is a continued elevated amplitude in the biopotential signals related tothe lifted finger, without further substantial wrist motion. The holdcan be considered detected upon detection of two or more windows ofelevated amplitude in the biopotential signal related to the liftedfinger. The hold can be considered ended when the biopotential signalrelated to the lifted finger drops significantly, which represents thefourth set of features defining the hold.

A primitive gesture called a “finger swipe” herein occurs when a fingeris lifted (460) and in the “hold” pose (460), and then the arm is movedsuch that the location of the wrist changes with a sweeping motion in adirection, such as to the right or to the left as shown at 462, or up ordown as shown at 464. Thus, the state of or change in the signals in oneor more windows immediately after the onset (the third set of features)is a change in the location of the wrist, in an amount over a threshold,in a given direction. The figure swipe can be considered detected upondetection of the wrist motion after detection of the lift and holdposes, without involving detection of a fourth set of features toconsider the finger swipe ended. A primitive gesture similar to a fingerswipe is a combination of a lift and hold pose followed by a rotation ofthe top of the wrist in a clockwise or counterclockwise direction asshown at 466.

A primitive gesture called an “analog lift” herein occurs when a fingeris slowly lifted from an initial pose 452 to the lifted pose 454 over aperiod of time. The amplitude of the biopotential signal typically iselevated for an analog lift, in comparison to a lack of movement, yet isinitially lower than when the biopotential signal indicates a lift, butalso gradually increases over a period of time. Thus, the state of orchange in the signals in one or more windows immediately after the onset(the third set of features) is a continued increase in the biopotentialsignal related to the lifted finger. The analog lift can be considereddetected upon detection of two or more windows of successive change,whether increase or decrease, in the biopotential signal related to thelifted finger. The analog lift can be considered ended when thebiopotential signal related to the lifted finger returns to its initialstate, which represents the fourth set of features defining the analoglift. The analog lift can thus be used as an input to slowly increase ordecrease a value while the analog lift continues over the period oftime.

Because the output of the location sensor (108 in FIG. 1A), when locatedat the top of the wrist, is related to location of the wrist, thelocation signal can be a primary source of information about position,orientation, and any changes in these, of the top of the wrist. It ispossible to distinguish, using the location signal, different positions,orientations, and motion of the top of the wrist. The locations andchanges of location of the wrist can be interpreted as a variety ofprimitive gestures.

Two illustrative examples of primitive gestures based on location willnow be described: relative location and rotations. A relative locationcan be based on position along one or more axes. In the examples below,relative locations are defined with reference to a grid of two or moreelements in one or two or three dimensions to which position is mapped.In the examples below, a grid is described has having three verticalelements and three horizontal elements. Similarly, rotation can be basedon orientation. In the example below, rotations are expressed as adegree of rotation of the wrist with respect to the elbow in two or moresubdivisions of 180 degrees. In the examples below, the subdivisions aredescribed as a positive or negative N-degree rotation (from +90 to -90degrees). A variety of other primitive gestures primarily related tolocation can be defined based on the location signal, whether alone orin combination with other signals.

An orientation, or rotation, of the wrist and hand with respect to theelbow can be detected, for example based on “roll” detected by agyroscopic sensor located at the wrist. For example, as shown in FIG.4A, the wrist may be in a “neutral” position 452, or in a 90 degreeclockwise rotation, or “palm up” position 470, or in a 90 degreecounter-clockwise rotation, or “palm down” position 472, or can be inthe process of being rotated from one orientation to another. The use ofthree different relative rotational locations (orientations) is merelyillustrative; any number of different relative rotational locations canbe used depending on granularity and reliability of detection of therotation. The amount of rotation can be represented by a N-degreerotation.

The orientation of the hand with respect to the wrist also may bedetected (the amount of bend of the hand with respect to the forearm atthe wrist). For example, the hand orientation can in an “up” position,“down” position, or “neutral” position with respect to the wrist. Theuse of three different relative orientations is merely illustrative; anynumber of different relative orientations can be used depending ongranularity and reliability of detection of the orientation. Theorientation can be represented by, for example, a N-degree offset fromneutral, or a classification of the direction, or a direction. Rotationof the hand to the left or right with respect to the wrist also may bedetected.

A relative vertical location can be detected based on location at thebeginning of, end of, or during vertical motion of the arm, such as bylifting or lowering of the arm. For example, as shown in FIG. 4B, thewrist may be “high” or “up”, such as above the shoulder at 480; “low” or“down”, such as below the hips at 482; or “medium” or “neutral”, such asbetween the hips and shoulders at 484, or can be in the process ofmoving from one location to another. Using the shoulders and hips asthresholds are merely illustrative; other threshold heights or referencepoints can be used. For example, one may determine the height of thewrist with respect to the elbow instead of the shoulder. Also, the useof three different relative heights is merely illustrative; any numberof different relative vertical locations can be used depending ongranularity and reliability of detection of the location. The directioncan be represented by a radial direction with respect to an anchorpoint, such as the elbow or waist or hips. The information can beprovided as an angle, which can be representative of an arm angle whenthe user interface device is worn on the arm.

A relative horizontal location similarly can be detected based onlocation at the beginning of, end of, or during horizontal motion of thearm, such as by sweeping the arm left or right. For example, the wristmay be to the wearer’s “left”, “middle”, or “right”, or can be in theprocess of moving from one such location to another. The use of threedifferent relative horizontal locations is merely illustrative; anynumber of different relative horizontal locations can be used dependingon granularity and reliability of detection of the location. Thedirection can be represented by a radial direction with respect to ananchor point, such as the elbow or spine. In some implementations, thehorizontal direction can be provided for example by a geomagneticsensor. In some implementations, the horizontal direction can be derivedfrom accelerometer or gyroscopic data from an initial referenceposition.

An application can include a graphical user interface that allows a userto change the definitions of relative locations. For example, a gridwith a number of grid elements can be defined based on thresholds usedto define relative vertical and horizontal locations. A graphical userinterface can present a graphical representation of this grid to theuser. The user can manipulate the number and sizes of grid elements thatdefine relative horizontal or vertical locations. For example, as shownin FIGS. 6 and 8 (described in more detail below), if displayed on atouchscreen or other conventional graphical user interface allowingselection of displayed objects, lines defining a grid can be selectedand moved by the user. In response to a user selection, the applicationon the responsive device updates the dimensions of the grid elements.The updated grid elements can be displayed interactively during thisoperation. When complete, the updated grid elements can be used toprocess the wrist location information into the relative vertical andhorizontal location.

As indicated above, other primitive gestures that can be detected arebased on one or more poses or locations being detected at a moment intime, or in a sequence over time, or as co-occurring at a moment intime. One kind of primitive gesture that is particularly useful is aprimitive gesture intended to cause the responsive device to startprocessing signals from the user interface device, or from another inputdevice or sensor, or both. Such a primitive gesture is herein called a“wake word”. In some implementations, prior to processing signals fromthe user interface device, the user interface device can be set in astate in which it is not transmitting data from sensors other than thesensor(s) that provide(s) signals from which the primitive gesturesdefining the wake word are detected. The wake word can be used to“unlock” the user interface device and start processing data receivedfrom the user interface device. Similarly, the same wake word, oranother primitive gesture (a “sleep word”), can be used to “lock” theuser interface device and stop processing data received from it. Such awake word or sleep word can control processing performed by theresponsive device, or can be specific to an application running on theresponsive device, or both.

The wake word or the sleep word can be any sequence of primitivegestures that can be detected from the sensors that are activelymonitored. Ideally, the sequence of primitive gestures defining the wakeword are unlikely to naturally occur from ordinary motion of the wristor arm or movements of the hand or fingers; instead this sequence ofprimitive gestures should be likely to occur only intentionally by theperson. As an example, rotating the wrist in a sequence of specificN-degree rotations (described below) can be used as a wake word.

In some implementations, the sequence of primitive gestures defining thewake word or the sleep word can be processed by a low-level driver ofthe operating system of the responsive device as a signal to activate ordeactivate the responsive device’s processing of data received from theuser interface device, or from another sensor or input device. In someimplementations, the sequence of primitive gestures defining the wakeword or the sleep word can be processed by an application as a signal toactivate or deactivate the application’s processing of inputs based onthe responsive device processing the data received from the userinterface device. In some implementations, the sequence of primitivegestures defining the wake word or the sleep word can be processed onthe user interface device to activate or deactivate transmission of datafrom the user interface device to the responsive device. In such animplementation, the responsive device may be configured to continuallyprocess data received from the user interface device. In someimplementations, the wake word or sleep word can activate processing ofanother form of input device or sensor data, such as to activate voicerecognition applied to audio signals received from a microphone on theuser interface device, responsive device, or other device.

Where a wake word or a sleep word for unlocking and locking the userinterface device both are processed by a low level device driver for theoperating system of the responsive device, in some implementations, theresponsive device can receive signals from the user interface devicethrough an input port of the responsive device. A device driver for thatinput port can process the received signals to determine whether anunlock command or a lock command has been received. In response to theunlock command being received, subsequent data can be passed to theoperating system of the responsive device for further processing by theoperating system or other applications managed by the operating system.Similarly, in response to the lock command being received, the devicedriver can stop passing data from the user interface device to theoperating system of the responsive device.

Referring now to FIG. 4C, a state diagram is used to illustrate theoperation of detecting primitive gestures. Typically, the data to beprocessed is in the form of a sequence of digital samples having anamplitude for each sample and a sampling rate. The system defines asequence of “windows” of samples. A window of samples can be specifiedby a. a moment in time in the signal and b. a period of time or numberof samples around that moment in time. For each window, features arecalculated based on the samples in that window, using digital signalprocessing techniques. For each window, the sampled data and anyfeatures extracted from the sampled data in that window, are processedto determine if features related to a primitive gesture have beendetected in that window. Any given window, or a collection of windows,can be labeled as representing a primitive gesture or not representing aprimitive gesture. Information about features detected in previouslyprocessed windows can be used to determine if a primitive gesture hasbeen detected in a current window. More details about how windows can beprocessed are provided below. Processing the received data as windows ofsamples and corresponding features allows a state-based and sequentialapproach to processing the received data to detect occurrence ofprimitive gestures, combinations of primitive gestures, and sequences ofprimitive gestures. An example implementation of this state-basedprocessing is in FIG. 4C.

Processing begins in an initial state 400 prior to any wake word havingbeen received from the user interface device. In response to data otherthan the wake word, as indicated at 402, processing remains in thisinitial state 400. In response to detecting a wake word, processingtransitions to state 404, as indicated at 406. A wake word received inthis state can be used to transition 440 back to the initial state 400.

Within state 404, processing primarily monitors the information from thelocation sensor on the wrist. So long as the information from thelocation sensor on the wrist indicates substantial motion, processingremains in this state 404, as indicated at 408, effectively implementingmotion silencing. While in this state, information about the positionand orientation of the wrist can be updated based on the informationfrom the location sensor. If there is no substantial motion of thewrist, the information from the biopotential sensor can be analyzed todetermine if there is any indication of movement of the hand or fingers,such as a finger lift. In a finger lift is detected, a transition to alift state 410 occurs, as indicated at 412. Similar states could beprovided for detection of other features related to other movements.

In the lift state 410, processing determines whether the detectedmovement indicates a lift, an analog lift, or a hold, and which of thoseis indicated, by analyzing the subsequently received data. If thedetected movement indicates a lift, an operation can be performed inresponse to the lift and a transition 414 back to state 404 can occur.As an example, the operation performed can be based on the currentposition and orientation of the wrist at the time the lift was detected.

If the detected movement indicates an analog lift, a transition 416 toan analog lift state 418 occurs. In this state, an operation responsiveto the analog lift can occur while the analog lift continues to bedetected, as indicated at 420. The operation performed can be based onthe current position and orientation of the wrist at the time the analoglift was detected. When the analog lift ends, a transition 422 back tostate 404 can occur.

If the detected movement indicates a hold (the biopotential sensordetects a continued level of activation of the finger muscle), atransition 424 to a hold state 426 occurs. In this state, an operationresponsive to the hold can occur while the hold continues to be detectedand there is no motion of the wrist detected, as indicated at 428. Whenthe hold ceases to be detected, a transition 432 back to state 404 canoccur. If motion of the wrist is detected while the hold also continuesto be detected, a transition 430 occurs to, and a finger swipe isindicated as detected in, state 434 and a corresponding operation can beperformed. After the operation related to the finger swipe is performed,a transition 436 back to state 404 can occur. The operation performed instate 428 or 434 can be based on the current position and orientation ofthe wrist at the time the hold or finger swipe was detected.

Referring now to FIG. 5 , a data flow diagram illustrating behavior ofthe responsive device (200 in FIGS. 2A-2D), in response to data (202)received based on signals from the sensors of the user interface device(100 in FIG. 1A), to provide inputs to applications, will now bedescribed. The responsive device, after being activated to receive andprocess information from the user interface device, can performoperations based on information 502, which is derived from location ofthe wrist as detected by the location sensor, and based on information504, which is derived from biopotentials related to muscle movement asdetected by the biopotential sensor, and combinations of the two. Theinformation 502 and 504 can be any of the digital samples, features,pose, location, or detected primitive gestures and state informationproduced by any of the layers of processing of data from the userinterface device, along with any other available context information. Insome implementations, the responsive device can include a selectionmodule 706 which can select an application (508-1, 508-2, ..., 508-N).Within an application (508-1, 508-2, ..., 508-N), the applicationperforms an action based on the information 510, which can include theinformation 502 or 504 and any additional information derived from theseby the responsive device or the application.

An application can select an operation based on one or more of thedetected location of the wrist or the detected pose of the hand, orcombination of these. In some implementations, the responsive device canassociate actions with locations of the wrist, and those actions can beactivated in response to a pose of the hand being detected while thewrist is in a given location. As described above, settings that definethresholds for determining relative wrist locations can be manipulatedin a graphical user interface. Such settings also can includeassociating, with a location, one or more actions with a pose or otherprimitive gesture that may occur while the wrist is in the location.

Several additional examples will now be described to further illustratethis kind of behavior by the responsive device.

First Example

In one example implementation different applications are assigned todifferent locations of the wrist with respect to the body. For example,a first application can be assigned to an “up” position, a secondapplication can be assigned to a “down” position, and a thirdapplication can be assigned to a “neutral” position. The responsivedevice, while it has determined that the user interface device is in theup position, directs inputs to the first application. The responsivedevice, while it has determined that the user interface device is in thedown position, directs inputs to the second application. The responsivedevice, while it has determined that the user interface device is in theneutral position, directs inputs to the third application.

The inputs directed to the applications can be, for example, based onone or more primitive gestures detected while the user interface deviceis in the given position. For example, the user may perform a lift withthe index finger while the user interface device is in the up position.An input based on this primitive gesture is then passed to the firstapplication. As another example, the user may perform an N-degree wristrotation while the user interface device is in the down position. Aninput based on this primitive gesture is then passed to the secondapplication. As another example, the user may perform an N-degree wristrotation followed by a lift with the index finger while the userinterface device is in the neutral position. An input based on thissequence of primitive gestures is passed to the third application.

As a specific illustrative use case, a person may be walking whilecarrying a mobile computing device (a responsive device), such as acellular phone, while wearing a wristband (a user interface device) withan inertial measurement unit and biopotential sensors. The mobilecomputing device may be in the person’s pocket, or held in the person’shand, or may be wrist-worn, or elsewhere on the person’s body. Themobile computing device and the user interface device may be combined inthe same device. The mobile computing device may have a display, butthis display may or may not be visible to the person. The mobilecomputing device typically has an operating system, such as the iOSoperating system or Android operating system, and can be running one ormore applications. For example, a first application can be a weatherapplication, a second application can be a music playback application,and a third application can be a map application. The person may bewearing headphones that communicate with the mobile computing device, orthe mobile computing device may have a speaker, to playback audio underthe direction and control of the mobile computing device.

Using this example setup, the person may raise her hand above the head,and then perform an index lift. In this case, the mobile computingdevice directs the index lift to the weather application, which in turnmay cause the mobile computing device, for example, to play audiothrough the headphones using is a speech-generated version or recordingof a current weather forecast.

The person may have her arm, wrist, and hand in the neutral position,and perform an index finger lift. In this case, the mobile computingdevice directs the index lift to the audio playback application, whichin turn may cause the audio playback application to begin playing backmusic. The current music selection to be played back may be determinedby various default settings. After playback begins, the audio playbackapplication can be attentive to further inputs, which would providevarious playback controls (see the second example below).

The person may lower her hand below her waist or hips, to a lowposition, and perform an index finger lift. In this case, the mobilecomputing device directs the index lift to the map application, which inturn may cause the mobile computing device to, for example, play audiothrough the headphones using a speech-generated version or recording ofa description of the user’s current location on the map.

In this example implementation, the operating system of, or a controlapplication running on, the mobile computing device can include theselection module 506 in FIG. 5 . The selection module can include amapping of detected wrist locations to applications. For example, theselection module can maintain a data structure, such as a table, thatstores data representing a relationship of a detected wrist location toa corresponding application. In response to receiving data indicating adetected wrist location from the user interface device, the operatingsystem or a control application can select the application correspondingto that wrist location as the target of any action associated with adetected primitive gesture based on the data received from the userinterface device.

Second Example

As another example implementation, consider an application in whichmultiple functions can be selected, such as an audio playbackapplication. Illustrative examples of audio playback operations includeplay, stop, pause, skip forward, skip backward, select a playlist,select an item in a playlist and so on.

In this example implementation, different operations can be assigned todifferent elements in an M x N grid, with a minimum size grid of 1 x 2or 2 x 1, i.e., a minimum of two grid elements. The grid can berectangular, with grid elements within the grid can be rectangular,uniform, and adjacent. The grid elements can be non-rectangular. Thegrid elements can be non-uniform. The grid elements can be non-adjacent.The grid elements may have different shapes and sizes. The grid elementscan be arranged in two dimensions or three dimensions. The gridelements’ shapes and sizes can be user-defined and user-modifiable, orcan be defined by the computer system and can be fixed, or a combinationof the two. Each element in the grid can be assigned to one or moreoperations to be performed. The application can include multiple grids,where in response to a primitive gesture occurring while one grid isactive, results in the application activating another grid forprocessing further primitive gestures.

Given a currently active application and a currently active set ofoperations, the detected location of the wrist, optionally incombination with one or more detected poses or primitive gestures, canbe passed to the currently active application. The currently activeapplication in turn performs one or more operations based on at leastthe detected location of the wrist. The application uses the detectedlocation to select a grid element of the currently active grid, and thenperforms an operation associated with that grid element. The operationperformed may depend on a detected pose or primitive gesture.

The grid representing the set of operations and its mapping to detectedlocations may be invisible to the user. However, although not visible tothe user, the grid or grids are conceptually or virtually known orunderstood by the user so that the user can select locations that theuser knows will be interpreted as associated with corresponding ones ofthe elements of the grid. In some applications the grid may be visibleon a display.

As a specific illustrative use case, consider an audio playbackapplication for which various operations are provided, such as start,stop, and pause playback, skip forward, skip backward, volume up, andvolume down. Referring to FIG. 6 , an example grid representing aconceptual spatial mapping of the operations to locations is shown. Inthis example, start, stop, and pause are mapped to a center region 600.Skip forward is mapped to a region 602 to the right; skip backward ismapped to a region 604 to the left. Volume up is mapped to a region 606at the top; volume down is mapped to a region 608 at the bottom. Thismapping of operations to locations can be explained to the user in somemanner so that the user can understand conceptually where the differentregions are located, and the operations associated with them. On adevice with a display, a visualization of this mapping can be presentedon the display.

While the audio application is the active application for receivinginputs, the responsive device can provide inputs to the audioapplication based on detected primitive gestures. In one exampleimplementation, an input can activate a spatial (virtual or “real”)menu, such as in FIG. 6 , in response to a primitive gesture beingdetected. While the index finger is lifted, motion of the wrist (i.e.,performing a finger swipe) allows other operations to be performed. Forexample, a finger swipe to the right or left allows the skip forward orskip backward operations to be selected. A finger swipe up or downallows the volume up or volume down operations to be selected. Rotationof the wrist, by rotating the forearm about the elbow, can be used toprovide additional input. The processing of operations with respect tomotion of the wrist can end when the user stops holding the index fingerin the lifted position.

As an example, a user can have the wrist at a neutral position withrespect to the body, perform an index finger lift, hold the index fingerin the lifted position while moving the wrist to the right, and thenrelease the index finger lift. In response to this combination of wristlocations and poses of the hand, the responsive device directs the wristlocation and pose information to the audio application. In turn, theaudio application activates the grid 800, and processes the wristlocation and pose information as an instruction from the user to skip toa next song on a playlist. The audio application begins play back of thenext song in the playlist. After user stops lifting the index finger,the audio application stops processing inputs with respect to the gridand continues playing back the selected song. As another example, ananalog lift can be used to gradually increase or decrease volume ofsound being played on a device.

An application can maintain state information based on the informationreceived based on the user interface device. Referring now to FIG. 7 ,the application can be in a first listening state 700, in which a firstprimitive gesture, such as an index finger lift and hold, can activatemonitoring for further inputs related to multiple grid elements of afirst grid. The first primitive gesture invokes a transition 702 to asecond state 704. Motion of the wrist detected after that firstprimitive gesture, in this second state, can be used to determine aspatial position, which in turn is mapped to one or more grid elements,called the selected grid element. A particular state (e.g., 706, 708)can be assigned for each grid element. In some implementations, whilethe wrist remains in a spatial position corresponding to the selectedgrid element, e.g., in state 708, an operation related to the selectedgrid element can be performed, as indicated at 710, such as increasingthe volume. The application can remain in this state until the usermoves the wrist location to be no longer on this grid element, or untilthe grid or application is no longer activated.

In some implementations, while the wrist is in a spatial positioncorresponding to the selected grid element, e.g., in state 706, anoperation related to the selected grid element can be processed afteryet another, second primitive gesture occurs as indicated by transition712 to state 714. Thus, the application can be placed in state 714 bythe user placing the wrist at a neutral vertical location (correspondingto the audio application which is in state 700), performing an indexfinger lift (activating the grid of commands and transitioning to state704), swiping the wrist up (causing a transition to state 706), and thenperforming another action, such as an N-degree rotation, (causing thetransition to state 714). After processing the operation for this otherstate 714, a transition to another state can occur, such as a transitionto state 704 or 700.

Motion of the wrist while the index finger is lifted can cause theapplication to switch its selection among the different grid elements,by switching between states, for example between states 706 and 708. Forexample, after a user initially moves the wrist to the right and theapplication initially selects the grid element to the right, a user maymove the wrist to the left, causing the application to select a gridelement to the left. A third primitive gesture, such as ending an indexfinger lift and hold, can terminate the processing of motion informationfor the wrist in connection with the spatial elements, causing atransition back to the initial state 700.

Third Example

As another illustrative use case, consider a controller for a miniature,unmanned, airborne vehicle, i.e., drone, which has a controllerresponsive to commands received wirelessly over radio frequencies tocontrol the motion of the vehicle. For such a vehicle, various controloperations can be provided for controlling motion, such as to direct itto move up or down, hover, rotate left or right, or move forward orbackward, or some combination of these. Such control is typicallyreferred to as “low-level” control. A wake word can be used to initiatesuch low level control through the user interface device.

Referring to FIG. 8 , an example grid representing a spatial mapping ofwrist locations to operations is shown. In this example, the absence ofwrist motion is mapped to a hover operation, as indicated at a centerregion 800. Motion of the wrist to the right, mapped to a region 802 onthe middle right, is mapped to moving the drone to the right. Motion ofthe wrist to the left, mapped to a region 804 on the middle left, ismapped to moving the drone to the left. Motion of the wrist up, mappedto a region 806 in the top center, is mapped to moving the drone up orforward. Motion of the wrist down, mapped to a region 808 in the bottomcenter, is mapped to moving the drone down or backward. The distinctionbetween up/forward or down/backward can be implemented based on detectedmovement of the finger. For example, a hold in combination with an up ordown wrist location can signal moving the drone up or down; whereas, theabsence of a hold can cause the drone to move forward or backward. Theupper left and right, and lower left and right corners can indicate acombined motion of the wrist, or can be mapped to another command. WristN-degree rotations (not shown in FIG. 8 ) clockwise or counterclockwisealso are wrist motions that can be mapped to corresponding rotations ofthe drone in a corresponding direction and amount.

Examples Using Geographic Location

In some implementations, a responsive device can use information basedon the data received from one or more user interface devices todetermine a geographic location. In some implementations, thedetermination is assisted using additional sensor information oradditional context information, or both, received from one or more otheruser interface devices, or one or more other sensors. or other source ofinformation, or any combination of these.

Some example scenarios in which geographic location can be determinedinclude, but are not limited to, the following.

An individual may be pointing in a specific direction, such as north, orsouth, and may want information about what is in that general direction.An individual may point at a street corner or a building and may wantinformation about it, or simply to identify it. An individual may pointin a direction and intend to set a point on a map. For example, anindividual may point at their feet to set the point on the map or tocommunicate their location to another device. An individual may point ina direction to identify an object, such as to spot a deer or otheranimal while hunting, or for communicating during a game or otheractivity.

Multiple individuals can be pointing at roughly the same location or asame object. Multiple users may want to cooperate to select or identifythe same object. Multiple users may want to cooperate to select oridentify multiple objects. The object or objects may be moving. Forexample, multiple firefighters can be pointing at a same window or otherposition on a building. Multiple individuals on a street can be pointingat the same vehicle on the street. For example, several police officerscan be pointing at the same car. Multiple individuals in a venue may bepointing at roughly the same location. Multiple people in a city parkmay be pointing at an object. For example, security personnel may bepointing at the same person, or at an open door, or at an object.Multiple people on a search and rescue mission may identify an object,or a person or an animal to be rescued, or a hazard or obstacle.

The selected geographic location can be used to control communication.For example, an individual can point at another individual and initiatecommunication directly with that individual. An individual can select agroup of individuals, such as by pointing at them individually orselecting them as a group with a gesture, to communicate with the group.An individual can, with a gesture such as pointing to the sky, initiatecommunication with a predetermined set of individuals. Similarly,communication with, or control of, or both, objects can be performed bypointing at an individual object, selecting a group of objects within aset, or selecting a predetermined set of objects.

The geographic location can be defined as an absolute location or arelative location. Examples of absolute locations include, but are notlimited to, coordinates in a map system, geographical positioning system(GPS) coordinates, or longitude and latitude in various resolutions, orany combination of these. Examples of relative locations include, butare not limited to, a reference point and a direction, or a referencepoint and a direction and a distance, a plurality of reference pointsand respective directions from those reference points, or a plurality ofreference points and respective directions and distances from thosereference points, or any combination of these.

The geographic location can be determined based on inputs from sensordata from a sensor associated with at least one person. Additionalsensor data from sensors associated from one or more other persons, orone or more other objects, or a combination of these, can be used todetermine the geographic location. Examples of such sensors include atleast any one or more of the following: a location sensor (as describedbelow) for locating a body part, a geomagnetic sensor, a compass, amagnetometer, an accelerometer, a gyroscope, altimeter, eye trackingdevice, an encoder providing information indicating a change inposition, a camera providing image data or motion video, a reference toa known landmark, or other sensor that can provide an indication of adirection from a person, or any combination of one or more of these. Thesensor data can be interpreted as, for example, any of a number ofprimitive gestures by the person, an intended action, an arm angle orother indication of relative vertical location, a direction or bearing,a distance, or an altitude or any combination of these. The geographiclocation can be determined based on such sensor data received fromsensors associated with one or more persons.

In some implementations, the determination of the geographic locationcan be assisted using context data. Such context data can includeinformation, such as a geographic location, of one or more objects,voice, sound, or text data, image data from one or more cameras, mapdata or user inputs with respect to a map, or any combinations of theseor yet other context information.

In some implementations, additional information, such as such contextdata, can be used to provide inputs to select, control, actuate, orperform other actions related to a geographic locations or an objectselected based on the geographic location. For example, voice, gesture,or other data can be used as a modifier. For example, a firefighterpointing at a building can communicate through voice, e.g., “two personsare near the second floor window!”. For example, voice, gesture, orother data can be used to annotate a selected object, with differentgestures being used to provide different labels for the object, such as“safe” or “dangerous”. For example, voice, gesture, or other data can beused to activate a command with respect to an object, with differentactions being actuated in different ways. Such actuations can beanalogous to a “click”, but can be obtained from any other data receivedfrom a user.

FIG. 9A is an illustrative data flow diagram of a system that determinesgeographic location. In FIG. 9A, received data 950-1, 950-2, ...,hereinafter “950”, from one or more user interface devices, is receivedat a responsive device 952. In some implementations, received data 950-2from a second user interface device (if available) may be communicatedthrough its own responsive device 954, which in turn providesinformation to the responsive device 952. In some implementations,responsive device 952 and 954 can provide information to yet anotherdevice to determine the geographic location.

To assist in determining geographic locations based on the data receivedfrom the user interface devices, the received data 950 includes at leastsensor data from one or more sensors on a user interface device.Examples of such sensors include at least any one or more of thefollowing: a location sensor (as described below) for locating a bodypart, a geomagnetic sensor, a compass, a magnetometer, an accelerometer,a gyroscope, an altimeter, an eye tracking device, encoders providinginformation indicating a change in position, image data from cameras, areference to a known landmark, or sound sensor, such as one or moremicrophones, or other sensor that can provide an indication of adirection from a person, or any combination of two or more of these. Insome implementations, the responsive device 952 can use the receiveddata 950 from the user interface device to determine a pose of the handand motion of the wrist, from which information such as pointing, apointing direction, pointing with motion can be determined. Otherprimitive gestures also can be detected to indicate selection of anobject or an action to be performed with respect to a selected object.

The responsive device 952 also may receive other context data 956. Theother context data 956 can be any one or more of at least the followingtypes of data.

The context data can include information, such as a geographic location,of one or more objects. For example, a database of information aboutobjects can be accessed.

The context data 956 can include voice or other audio data from one ormore microphones. The context data 956 can include text or other dataderived from speech or other audio signals recognized from such audiodata.

The context data 956 can include image data or video data from one ormore cameras. The video data can provide depth information provided bytime of flight processing. Similarly, other time of flight sensors canprovide such depth information. The context data 956 can include resultsof object recognition or surface detection based on such image data.

The context data 956 can include user input with respect to a map orother information associated with a map. Such user input can include,for example, a location, area, or curve with respect to the map. Pointof interest data associated with a map of the area around the user(s)also can be available as context data 956.

Turning now to FIGS. 9B and 9C, and example use case in which ageographic location can be determined is illustrated. In FIG. 9B, anarea 980 is shown in which three persons 982, 984, and 986 areillustrated at different geographic locations within the area 980. Thedrawing is not to scale, but is intended to show some perspective ordepth with the individual each standing on the ground within the area980. Each person is pointing at an object 988, illustrated above ground.It should be understood that individuals providing such information canbe located anywhere, and not necessarily on the ground, as thesetechniques can apply to people on the ground, in vehicles, under watersuch as for scuba diving, or in the air, such as in airborne vehiclessuch as airplanes or even parachuting. Each of the persons isillustrated at pointing at the object 988, thus providing differentdirections of pointing from their respective locations within the area980. An angle of an arm of a person also can be provided. From theinformation available from the individuals, namely, their locations anddirections of pointing, and optionally arm angles, a geographic locationof the object 988 can be determined. The location in two-dimensions,corresponding to the ground of area 980, can be determined using thelocations and pointing directions of the individuals. An elevation ofthe object can be determined further using the arm angles from theindividuals.

In FIG. 9C, an illustration of the geographic location of the object 988on an example display in two-dimensions is shown. For example, one ofthe persons can have a device with a display that provides a graphicalindication 990 of the determine geographic location of the object 988.The display optionally can include the geographic locations of theperson, e.g., P1. The display optionally can include the geographiclocations of the other persons, e.g., P2 and P3.

Returning to FIG. 9A, given the received data 950, and any context data956, the responsive device can determine the geographic location 960.There are many ways in which to compute the geographic location. Severalexamples follow.

The geographic location can be determined from the sensor data, andoptionally context data, using a variety of different kinds ofprocessing techniques. In some implementations, forms of triangulationcan be applied using directions from multiple reference points. In someimplementations, an input can allow a user to specify a distance inrelationship to a determined location and direction. In someimplementations, techniques from computational geometry can be applied,such as variants of simultaneous localization and mapping (SLAM)algorithms which compute a location of an agent, where the geographiclocation to be determined is treated as the location of the agent to becomputed. In some implementations, forms of odometry can be used wherechange in position over time is determined based on sensor data. In someimplementations, visual odometry can be performed on images fromcameras. Any combination of such implementations also can be used.

The sensor data used to determine a geographic location also can have atime stamp or other indicia of time. In implementations in which sensordata associated with multiple users or devices is used, then the timeinformation can be used to eliminate or weight sensor data. For example,more recent data can be emphasized or used, and less recent data can bedeemphasized or discarded.

The determination of the geographic location also can include dataindicating a level of certainty, or validation, or confidence, or arange about the determined geographic location. For example, such datacan be based on the number of inputs used to determine the geographiclocation. As an example, if data is obtained from a larger number ofsources, such as a larger number of individuals pointing at a samegeneral area, then this data likely it implies increased validation ofthe object being the intended object or an increasing meaningfulness ofthe object, or both. As another example, the data can be based on theresolution of the available data used to compute the geographiclocation.

For example, given a first geographic location and a first direction fora first person, defining a first vector, and a second geographiclocation and a second direction for a second person, defining a secondvector, the intersection of the first and second vectors provides alocation. The first and second geographic locations, and first andsecond directions, can be based on a time of detection of a primitivegesture, such as a finger lift occurring while a person is pointing in adirection, from the respective user interface worn by the person. Inresponse to the primitive gesture, the data available around that timecan be used as one or more of the geographic locations or directions.Thus, two individuals can point at an object, and their information canbe used to determine a geographic location of that object.

As another example, given a geographic location of a person, a bearingand a distance can be derived from the data received from the userinterface device. The bearing can be derived from signals from ageomagnetic sensor or by left and right motion of the wrist or both. Thedistance can be based on up and down motion of the wrist. A vector to adetermined geographic location can be displayed to the user such as on amap on a display or in a heads up display to provide feedback to theuser about the determined geographic location. The responsive device canupdate that location in response to further input from the user.

As another example, given a geographic location of a person, a bearingand a distance can be derived from the data received from the userinterface device and from data received from another device carried bythe person. For example, the person wearing the user interface devicemay have a mobile device such as a phone, which has its own geomagneticsensor or other sensor, or sensors, from which a first direction can bederived. The person wearing the user interface device may have aheads-up display that includes eye tracking. The persons wearing theuser interface device may have a second user interface device worn on adifferent arm from a first user interface device. The first directioncan be computed so that it is intended to represent, for example, a gazedirection of the person, or a direction in which the person’s head orchest or hand is pointing. The user interface device also provides oneor more signals from which a second direction can be derived, such as abearing based on the geomagnetic sensor, indicating a direction in whicha hand is pointing. Vectors representing the first and seconddirections, projected to a common plane, and differences between originsfor the first and second directions, can be used to determine a point ofintersection, providing a distance to the geographic location from oneor both of the origins. A vector to a determined geographic location canbe displayed to the user such as on a map on a display or in a heads updisplay to provide feedback to the user about the determined geographiclocation. The responsive device can update that location in response tofurther input from the user.

As another example, given a geographic location of a person, adirection, and a selected object, time of flight data of the selectedobject can be used to determine distance information. A vector to adetermined geographic location, based on the geographic location of theperson, their direction, and the distance to the selected object, can bedisplayed to the user such as on a map on a display or in a heads updisplay to provide feedback to the user about the determined geographiclocation. The responsive device can update that location in response tofurther input from the user.

In some implementations, context data can be used to refine a computedgeographic location. For example, if a user says “bank”, then a locationcorresponding to a “bank” can be selected. As another example, an areaselected on a map can be used to ensure the determined geographiclocation is in that area.

The geographic location in turn can be used in many ways, examples ofwhich include but are not limited to the following and can includecombinations of two or more of the following. A geographic location canbe used as an input to an application, such as to identify or select anobject. Various operations can be performed by an application inconnection with a selected object, of which non-limiting examplesinclude invoking tracking of the object by camera, or obtaininginformation about the object from a database. A geographic location canbe data communicated to another user or another machine. A geographiclocation can be treated as context information when selecting an actionto be performed in combination with data based on the signals receivedfrom one or more user interface devices. A geographic location can beprovided as feedback to a user, such as by displaying a point on a map.

Given a geographic location 960, a variety of operations can beperformed with such information in addition to feedback to a user on adisplay on a map. For example, in some implementations, such point ofinterest data can be provided as feedback data along with any determinedgeographic location 960. Such a geographic location 960, along withother information derived from the received data from the user interfacedevice, can be provided to an application for processing. One or moreobjects near the determined location can be detected or tracked orselected. An operation can be performed in connection with a detected orselected object, such as invoking tracking of the object by camera, orobtaining information about the object from a database.

In one example implementation, consider an application in which theperson is viewing objects, possibly in a three-dimensional space. Theviewer may be wearing a virtual reality display or other immersivedisplay. The viewer may be viewing a conventional two-dimensionaldisplay. The user interface device enables the user to intuitivelyselect and perform actions with respect to displayed objects, even whilegrasping other objects.

Referring now to FIG. 9D, with a virtual scene being displayed, acomputer has a three-dimensional model of the scene which is beingrendered, represented by scene data 903. This scene data 903 can beprocessed, using scene data processor 900, to generate data representinga three-dimensional surface 902. Given a location 904 of the userinterface device or user in the scene, and a vector 906 emanating fromthat location, an intersection of that vector and the 3D surface can bedetermined. That point of intersection corresponds to an object in thethree-dimensional scene. An “object selector” 908 can process the 3Dsurface, vector, and location information to select an object 910.

The inputs 911 based on the signals from the user interface device canbe used to compute the vector 906 using a vector computing module 918.In some applications, the vector can be provided by the application,such as a direction of view of an avatar in a video game. Similarly, theinputs 911 based on the signals from the user interface device can beused to compute the location 904 using a location computing module 920.

Given a selected object 910, and inputs 911 based on signals from theuser interface device, an operation 914 can be performed with respect tothe selected object. An operation selection module 916 receives theinputs 911 and selected object 910 and determines the selected operation914 to be performed.

The application can have, for each object, a set of possible inputs andcorresponding actions. The inputs received based on the user interfacedevice can be one or more of the possible inputs associated with theselected object. In response to these inputs, the application can selectan operation to perform on the selected object.

As another example implementation, consider an application in which theperson is viewing real-world objects through a heads-up or augmentedreality display. The user interface device enables the user tointuitively select and perform actions with respect to the real-worldobjects, even while grasping other objects. This selection of areal-world object, and performance of an operation with respect to thatselected object, can be implemented in an application in a similarmanner as described above in connection with FIG. 9B. However, in thisinstance, there is no artificial scene data 903 from which the surfaceof objects is already defined in three dimensions. Instead, a responsivedevice may be receiving video information representing a real-worldlocation, such as a camera from a headset worn by the user. In thisinstance, the video information is processed to extract surfaceinformation to provide scene data 903.

An example user interface is shown in FIG. 10 . This interface can beprovided, for example, in a heads up display. In this image, the usercan see the wrist-worn user interface device 1000, and a line 1002displayed as emanating from the wrist-worn device to a selected object1004. An overhead view 1006 in two dimensions can be shown. Otherobjects that can be selected are indicated at 1008 and 1010. Any one ofthese objects can be selected based on location of the wrist, and thenanother primitive gesture, such as an index lift, can designate anoperation to be performed on the selected object, such as to retrieveinformation about that object. In this example, the objection 1004, 1008and 1010 can be drones or other objects that can then be controlledafter being selected using techniques described above.

In another example implementation, consider an application in which theperson is viewing real-world objects in a two-dimensional, third-personview, such as shown in FIG. 11 . For example, a display may present atwo-dimensional aerial map 1100, where the user’s position, or aposition of a selected object, also is noted on the map, such as byarrow 1102. Using an arrow allows for an orientation based ongeomagnetic sensing to be reflected in the display. The user’sgeographic location may be entered by the user, or may be determinedusing global positioning system (GPS) data, or other data suitable fordetermining such a geographic location, which may be provided by amobile phone or other mobile device which the user may have on theirperson. The GPS or other geographic location information may be providedby the responsive device, such as a tablet computer being used by theuser.

In such a user interface, in response to motion of the wrist, such aslifting and lowering the arm, the responsive device can change thedistance on the map between the position 1102 along a line to anothergeographic location 1104 remote from the user. The responsive device canuse direction or heading information from a geo-positioning sensor onthe user interface device or from another source to change the directionof that line. For example, the user can move the wrist left and right tochange the bearing information, and can move the wrist up and down tochange the distance.

After manipulating the point on the map by the heading and distance fromthe position 1102, a primitive gesture can be used to select an objectat that point and perform an operation with respect to the selectedobject, perhaps based on another primitive gesture. For example, a liftcan be used to initiate an operation with respect to a selected object.

As another example, in response to a finger swipe, the responsive devicecan define an arc based on an initial position 1104 at the beginning ofthe swipe, and an end position 1106 at the end of the swipe, given aninitial heading and distance. In response to another primitive gesture,the responsive device can allow selection of one or more objects in thearea defined by the points 1102, 1104, and 1106.

In the examples given above, in some applications there can be multipleuser interface devices. In some implementations, multiple user interfacedevices house different parts of the user interface device, such as thebiopotential sensor and the location sensor, and may be worn on the samebody part. In some implementations, multiple user interface devices areworn on different body parts. For example, a first user interface devicecan be worn at the top of the wrist of the right arm and a second userinterface device can be worn at the top of the wrist of the left arm.When multiple user interface devices are used, they can be generally thesame, e.g., having the same form factor, sensors, and feedback devices,but also can be different. For example, a user interface device worn atthe top of the wrist may be configured differently from a user interfacedevice worn at the ankle. In implementations using multiple userinterface devices, the selection of an application or the selection ofan operation to be performed by an application, or both, further can bebased on which user interface device is providing the data based onbiopotential signals or location signals.

Having now described several examples of interaction of a person with aresponsive device through inputs provided by a user interface device,further details of example implementations of signal processing anddetection of primitive gestures will now be provided.

To enable various primitive gestures to be detected, conditioned signalsfrom the sensors are further processed to detect patterns in thosesignals which correspond to the primitive gestures. In someimplementations, signals received from the biopotential sensor can beprocessed to classify a portion of the biopotential signals, occurringat a moment in time, into data indicating a pose of the hand occurred atthat moment in time. Signals received from the location sensor can beprocessed to classify a portion of the location signals at a moment intime into data indicating a location of the top of the wrist at thatmoment in time. In some implementations signals from both thebiopotential sensor and the location sensor are processed together toobtain data indicating poses of the hand and locations of the top of thewrist at moments in time. The data about poses of the hand and locationsof the wrist can be processed to allow detection of primitive gestures.In some implementations, features are extracted from the biopotentialsignals and location signals, and these features can be processeddirectly into data indicative of primitive gestures.

The signal from a sensor can be in the form of a sequence of digitalsamples having an amplitude for each sample and a sampling rate. Toperform various signal processing operations on the signal, the systemdefines a sequence of “windows” of samples. A window of samples can bespecified by a. a moment in time in the signal, and b. a period of timeor a number of samples around that moment in time. For each window,features are calculated using digital signal processing techniques. Anygiven window, or a collection of windows, can be labeled as representinga primitive gesture or not representing a primitive gesture.

As an example, illustrative implementation, a window can be 200milliseconds, with adjacent windows overlapping by fifty percent (50%)meaning that the samples forming the last 100 milliseconds of a firstwindow also are the samples forming the first 100 milliseconds of asecond window immediately following the first window in a sequence ofwindows. Different window structures can be used for data collection andlabeling than used for real time processing and classification ofsignals. For example, for real-time classification, a sliding window of200 milliseconds without overlap can be used. Both the size of thewindow and the overlap of adjacent windows can be tuned for a particularuser interface design and application.

Given windows of samples of a conditioned sensor signal, each window isprocessed to extract features, which in turn are used for defining anddiscriminating between different patterns related to the kinds ofinformation desired to be extracted from the sensor signals, such aslocation or motion of the wrist, or pose of the hand, or othercharacterization.

A processing module within a responsive device that performs featureextraction or gesture detection can implement a device interface toreceive the sampled sensor data from the user interface device, andimplements an application interface to provide features or other data toother applications within the responsive device. The features can beoutput as a stream of data indicating, for each window, featuresassociated with the window.

Features that can be derived from signals include, for example, featuresobtained from processing a time domain representation of the signal, andfeatures obtained from processing a frequency domain representation ofthe signal. Features that can be obtained from the time domainrepresentation of the signal include, but are not limited to, theHudgin’s time domain features.

Examples features that can be computed for each window are listed inTables I-V. Not all of the listed features are required; and there maybe some features not listed that also can be used:

Table I Base feature set for a single channel Mean absolute value Numberof zero crossings Number of slope sign changes Waveform length Willisonamplitude

Table II Expanded feature set for single channel (addition to Table I)Root mean squared Variance Log detector Myoelectric pulse rate Meansquare root Energy Sample entropy Mean frequency Maximum frequencyMedian frequency Variance of amplitudes Modified mean frequency Modifiedmedian frequency First spectral moment Frequency ratio Mean absolutevalue slope First autoregressive coefficient

Table II Example features derived from comparing two channels Energyratio Energy difference

Table III Features extracted from IMU channels Mean of gyroscope valueMean of accelerometer value Standard deviation of gyroscope valueStandard deviation of accelerometer value

Table IV Feature combinations involving the signals from all gyroscopechannels or from all accelerometer channels Mean velocity Maximumvelocity Minimum velocity Mean acceleration Maximum acceleration Minimumacceleration

With enough data, the signal itself can be fed into deep learning modelswhich will learn their own features.

Model Training

One technique for processing signals to detect patterns is to use aparameterized model for which the parameters are trained based onsignals corresponding to patterns which are known. Such signals are“labeled”, meaning there is data associated with the signal indicatingthe pattern which the signal is known to contain. A computer systemprocesses labeled data to generate a model through a process called“training”. The resulting model can detect, in a real-time signal from asensor, whether a particular pattern occurred in the real-time signalfrom the sensor.

One of the challenges with training such models is acquiring asubstantial amount of labeled data, called a training set. The trainingset ideally includes positively and negatively labeled samples. Apositively labeled sample is a sample that is labeled as representativeof a known characterization of the signal. A negatively labeled sampleis a sample that is known not to include a known characterization of thesignal. Having such negatively labeled samples in a training set allowsa model to be trained which can better distinguish intentional activityfrom everyday activity of the person.

There are several ways to obtain positively-labeled samples for atraining set. In one implementation, a plurality of user interfacedevices can be equipped to receive signals from a computer. In responseto those signals, the user interface device generates a prompt to theperson wearing the user interface device to perform a specified action.The user then performs the specified action. The user interface devicesends the sensor signals to the computer, which labels the signalaccording to the specified action.

More specifically, each window of samples in the received signal islabeled according to the selected motion. Given a set of samplescomprising several windows, a window is labeled as corresponding to, ornot corresponding to a pose of the hand or a wrist location or primitivegesture or any combination of these. Given a set of samples and a label,a window labeling component identifies an onset. Windows of samplesoccurring prior to a window with an onset are not labeled. The set ofsamples in windows that are not labeled prior to the detected onset canbe discarded for the purpose of training.

Another example method of gathering training data is to use anapplication running in a computing device with a touchscreen or otherdevice that can detect actual hand and finger position.

Model training typically is specific to a specific version of sensorsand signal processing performed on signals from those sensors. Multipleusers with devices of the same configuration (i.e., same sensors, sensorconfiguration, low level signal processing, and feature extractionhardware), can provide data for training a model for that configuration.However, given a number of labeled windows of data including theircorresponding features, and optionally additional sensor datacorresponding to those windows, conventional training algorithms can beapplied to models that have been adapted to accommodate the specifickinds of signals being processed.

Training can be performed to create a trained model for a single userusing labeled data from a single user, or to create a trained model formultiple users using labeled data from multiple users. Training of amodel for a single user can be performed on a computing deviceaccessible to the user interface device of that single user. Formultiple users, labeled data can be transferred to and stored with aserver computer connected over a computer network to a source of eachuser’s labeled data.

After training and testing a model, the trained model can be deployed. Auser’s device, whether a responsive device or user interface device,downloads the trained model from a server computer or from the computingdevice that trained the model.

Some features can emerge from training such models as having an impacton classification. For example, it has been discovered that the energyratio between first and second channels of the first sensor is verypredictive for separating thumb and index extension. In some cases, thisratio can also be used to filter out other “false positives” such aswrist extension, pinky extension, etc. There is a dependence betweensensor configuration and signal quality, on the one hand, and theability to distinguish between certain poses, locations, primitivegestures, or combinations of these, such as between thumb and indexextension in terms of energy ratio.

Having now described several example implementations, FIG. 12illustrates an example of a general purpose computing device with whichcan be used to implement a responsive device or other computer systemsused in connection with such responsive devices. This is only oneexample of a computer and is not intended to suggest any limitation asto the scope of use or functionality of such a computer. The systemdescribed above can be implemented in one or more computer programsexecuted on one or more such computers as shown in FIG. 12 .

FIG. 12 is a block diagram of a general-purpose computer which processescomputer program code using a processing system. Computer programs on ageneral-purpose computer typically include an operating system andapplications. The operating system is a computer program running on thecomputer that manages access to various resources of the computer by theapplications and the operating system. The various resources typicallyinclude memory, storage, communication interfaces, input devices andoutput devices.

Examples of such general-purpose computers include, but are not limitedto, larger computer systems such as server computers, databasecomputers, desktop computers, laptop and notebook computers, as well asmobile or handheld computing devices, such as a tablet computer, handheld computer, smart phone, media player, personal data assistant, audioor video recorder, or wearable computing device.

With reference to FIG. 12 , an example computer 1200 comprises aprocessing system including at least one processing unit 1202 and amemory 1204. The computer can have multiple processing units 1202 andmultiple devices implementing the memory 1204. A processing unit 1202can include one or more processing cores (not shown) that operateindependently of each other. Additional co-processing units, such asgraphics processing unit 1220, also can be present in the computer. Thememory 1204 may include volatile devices (such as dynamic random accessmemory (DRAM) or other random access memory device), and non-volatiledevices (such as a read-only memory, flash memory, and the like) or somecombination of the two, and optionally including any memory available ina processing device. Other memory such as dedicated memory or registersalso can reside in a processing unit. This configuration of memory isillustrated in FIG. 12 by dashed line 1204. The computer 1200 mayinclude additional storage (removable or non-removable) including, butnot limited to, magnetically-recorded or optically-recorded disks ortape. Such additional storage is illustrated in FIG. 12 by removablestorage 1208 and non-removable storage 1210. The various components inFIG. 12 typically are interconnected by an interconnection mechanism,such as one or more buses 1230.

A computer storage medium is any medium in which data can be stored inand retrieved from addressable physical storage locations by thecomputer. Computer storage media includes volatile and nonvolatilememory devices, and removable and non-removable storage devices. Memory1204, removable storage 1208 and non-removable storage 1210 are allexamples of computer storage media. Some examples of computer storagemedia are RAM, ROM, EEPROM, flash memory or other memory technology,CD-ROM, digital versatile disks (DVD) or other optically ormagneto-optically recorded storage device, magnetic cassettes, magnetictape, magnetic disk storage or other magnetic storage devices. Computerstorage media and communication media are mutually exclusive categoriesof media.

The computer 1200 may also include communications connection(s) 1212that allow the computer to communicate with other devices over acommunication medium. Communication media typically transmit computerprogram code, data structures, program modules or other data over awired or wireless substance by propagating a modulated data signal suchas a carrier wave or other transport mechanism over the substance. Theterm “modulated data signal” means a signal that has one or more of itscharacteristics set or changed in such a manner as to encode informationin the signal, thereby changing the configuration or state of thereceiving device of the signal. By way of example, and not limitation,communication media includes wired media such as a wired network ordirect-wired connection, and wireless media include any non-wiredcommunication media that allows propagation of signals, such asacoustic, electromagnetic, electrical, optical, infrared, radiofrequency and other signals. Communications connections 1212 aredevices, such as a network interface or radio transmitter, thatinterface with the communication media to transmit data over and receivedata from signals propagated through communication media.

The communications connections can include one or more radiotransmitters for telephonic communications over cellular telephonenetworks, or a wireless communication interface for wireless connectionto a computer network. For example, a cellular connection, a Wi-Ficonnection, a Bluetooth connection, and other connections may be presentin the computer. Such connections support communication with otherdevices, such as to support voice or data communications.

The computer 1200 may have various input device(s) 1214 such as variouspointer (whether single pointer or multi-pointer) devices, such as amouse, tablet and pen, touchpad and other touch-based input devices,stylus, image input devices, such as still and motion cameras, audioinput devices, such as a microphone. The compute may have various outputdevice(s) 1216 such as a display, speakers, printers, and so on, alsomay be included. These devices are well known in the art and need not bediscussed at length here.

The various storage 1210, communication connections 1212, output devices1216 and input devices 1214 can be integrated within a housing of thecomputer, or can be connected through various input/output interfacedevices on the computer, in which case the reference numbers 1210, 1212,1214 and 1216 can indicate either the interface for connection to adevice or the device itself as the case may be.

An operating system of the computer typically includes computerprograms, commonly called drivers, which manage access to the variousstorage 1210, communication connections 1212, output devices 1216 andinput devices 1214. Such access can include managing inputs from andoutputs to these devices. In the case of communication connections, theoperating system also may include one or more computer programs forimplementing communication protocols used to communicate informationbetween computers and devices through the communication connections1212.

Each component (which also may be called a “module” or “engine” or thelike), of a computer system and which operates on one or more computers,can be implemented as computer program code processed by the processingsystem(s) of one or more computers. Computer program code includescomputer-executable instructions or computer-interpreted instructions,such as program modules, which instructions are processed by aprocessing system of a computer. Such instructions define routines,programs, objects, components, data structures, and so on, that, whenprocessed by a processing system, instruct the processing system toperform operations on data or configure the processor or computer toimplement various components or data structures in computer storage. Adata structure is defined in a computer program and specifies how datais organized in computer storage, such as in a memory device or astorage device, so that the data can accessed, manipulated and stored bya processing system of a computer.

It should be understood that the subject matter defined in the appendedclaims is not necessarily limited to the specific implementationsdescribed above. The specific implementations described above aredisclosed as examples only.

What is claimed is:
 1. A system for gesture-based control, the systemcomprising: a wearable device configured to be worn on a wrist of aperson and comprising one or more sensors, the wearable device beingconfigured to obtain at least: physiological data, the physiologicaldata being configured to indicate a state of a person’s hand; and wristlocation data, the wrist location data comprising information relatingto a state of the person’s wrist; wherein the system is configured to:based on at least the physiological data, detect a first gestureindicating an intention to control an object; in response to detectingthe first gesture, transition to an object control state in whichchanges in the wrist location data and/or the physiological data areconfigured to control a movement of the object; while the system is inthe object control state: detect, based on the physiological data and/orthe wrist location data, a first motion of a body part of the person ina first motion direction; in response to detecting the first motion ofthe body part of the person in the first motion direction, generate afirst command to move the object in a first command direction that ismapped to the first motion direction; detect, based on the physiologicaldata and/or the wrist location data, a second motion of the body part ofthe person in a second motion direction that is different from the firstmotion direction; and in response to detecting the motion of the bodypart of the person in the second direction, generate a second command tomove the object in a second command direction that is mapped to thesecond motion direction, the second command direction being differentthan the first command direction; after generating the first and secondcommands to move the object in the first command direction and thesecond command direction, detect, based on at least the physiologicaldata, a second gesture indicating an intention to cease control of theobject; and in response to detecting the second gesture, transition outof the object control state.
 2. The system of claim 1, wherein the firstgesture comprises an index finger lift-and-hold.
 3. The system of claim2, wherein the second gesture comprises ending the index fingerlift-and-hold.
 4. The system of claim 1, wherein while the system is inthe object control state, the system is further configured to: detect,based on the physiological data and/or the wrist location data, that thebody part of the person is in a neutral position; in response todetecting that the body part of the person is in the neutral position,cause the object to remain stationary.
 5. The system of claim 1, whereinthe object is a virtual object.
 6. The system of claim 1, wherein theobject is an unmanned vehicle.
 7. The system of claim 1, wherein the oneor more sensors comprises a plurality of electrodes, and thephysiological data is based on electrical signals indicating activitiesof nerves and muscles in and around the wrist of the person.
 8. Thesystem of claim 1, wherein the system further comprises a responsivedevice, wherein the responsive device is configured to perform the stepsof detecting the first gesture, the second gesture, the first motion,and the second motion based on the physiological data and/or the wristobtained by the wearable device.
 9. The system of claim 1, wherein thedetecting the first gesture comprises: based on at least the wristlocation data, determine a directional vector intended by the person;based on at least the physiological data, detect a selection gestureindicating an intention to make a selection; and based on thedirectional vector and the detected selection gesture, select a target,the selected target being indicated by the directional vector asdetermined when the selection gesture is detected, wherein the target isor includes the object to be controlled in the object control state. 10.The system of claim 1, wherein the first movement direction and thefirst command direction are different directions relative to a body ofthe person.
 11. A method for gesture-based control, the method beingperformed using a system comprising a wearable device configured to beworn on a wrist of a person and comprising one or more sensors, themethod comprising: obtaining, using the wearable device, physiologicaldata, the physiological data being configured to indicate a state of aperson’s hand; obtaining, using the wearable device, wrist locationdata, the wrist location data comprising information relating to a stateof the person’s wrist; based on at least the physiological data,detecting a first gesture indicating an intention to control an object;in response to detecting the first gesture, transitioning the system toan object control state in which changes in the wrist location dataand/or the physiological data are configured to control a movement ofthe object; while the system is in the object control state: detecting,based on the physiological data and/or the wrist location data, a firstmotion of a body part of the person in a first motion direction; inresponse to detecting the first motion of the body part of the person inthe first motion direction, generating a first command to move theobject in a first command direction that is mapped to the first motiondirection; detecting, based on the physiological data and/or the wristlocation data, a second motion of the body part of the person in asecond motion direction that is different from the first motiondirection; and in response to detecting the motion of the body part ofthe person in the second direction, generating a second command to movethe object in a second command direction that is mapped to the secondmotion direction, the second command direction being different than thefirst command direction; and after generating the commands to move theobject in the first command direction and the second command direction,detecting, based on at least the physiological data, a second gestureindicating an intention to cease control of the object; and in responseto detecting the second gesture, transitioning the system out of theobject control state.
 12. The method of claim 11, wherein the firstgesture comprises an index finger lift-and-hold.
 13. The method of claim12, wherein the second gesture comprises ending the index fingerlift-and-hold.
 14. The method of claim 11, further comprising: while thesystem is in the object control state: detecting, based on thephysiological data and/or the wrist location data, that the body part ofthe person is in a neutral position; and in response to detecting thatthe body part of the person is in the neutral position, causing theobject to remain stationary.
 15. The method of claim 11, wherein theobject is a virtual object.
 16. The method of claim 11, wherein theobject is an unmanned vehicle.
 17. The method of claim 11, wherein theone or more sensors comprises a plurality of electrodes, and thephysiological data is based on electrical signals indicating activitiesof nerves and muscles in and around the wrist of the person.
 18. Themethod of claim 11, wherein the system further comprises a responsivedevice, wherein the responsive device is configured to perform the stepsof detecting the first gesture, the second gesture, the first motion,and the second motion based on the physiological data and/or the wristobtained by the wearable device.
 19. The method of claim 11, wherein thedetecting the first gesture comprises: based on at least the wristlocation data, determining a directional vector intended by the person;based on at least the physiological data, detecting a selection gestureindicating an intention to make a selection; and based on thedirectional vector and the detected selection gesture, selecting atarget, the selected target being indicated by the directional vector asdetermined when the selection gesture is detected, wherein the target isor includes the object to be controlled in the object control state. 20.The method of claim 11, wherein the first movement direction and thefirst command direction are different directions relative to a body ofthe person.